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BOUNDARY -SPLINE- WAVELET COMPRESSION FOR VIDEO IMAGES 



* * * * 



This invention is in the field of video image storage 
and communication, and is more specifically directed to 
compression and decompression of video images, including 
motion video images and high-quality still video images. 



Background o-F the Inve»ni--irm 

2^ Conventional vid eo eomnrf»Hai rm 

Modem electronic data processing systems are now 
often used not only for conventional numeric and text 
processing tasks, but also for processing data 
corresponding to visual images. As such, many modern 
computer and telecommunications systems have "multimedia" 
capabilities, where the data processed and communicated 
includes video images either generated by a computer or 
digitized from a conventional video camera. 

The capacity required to communicate video images on 
a real-time basis is huge, however, when measured against 
modern capabilities. For example, a single video image 
(i.e., frame) displayed by a rectangular array of picture 
elements ("pixels") arranged in 640 rows by 800 columns, 
with the color of each pixel represented by twenty- four 
digital bits, would require over 1.52 million bytes (1500 
kbytes) of digital memory to store all information in the 
frame. While this memory requirement for a single video 
frame is large, digital storage of a series of frames, such 
as a motion picture, will quickly consume the disk storage 
capacity of even high-end computers and workstations. 
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The large amount of digital data necessary to 
represent a video frame not only impacts computer storage 
requirements, but limits the rate at which conventional 
systems can communicate motion pictures. Considering that 
conventional high-speed digital communication channels have 
a bandwidth of on the order of from 4 0 to 80 MHz, it 
becomes quite apparent that conventional motion pictures of 
thirty frames per second, with 1500 kbytes of digital 
information per frame, cannot be fully transmitted in real- 
time over state of the art digital communications systems. 

In response to these limitations on the ability of 
modern computer systems to store and communicate video 
information, various types of data compression techniques 
have been developed in recent years. Conventional data 
compression techniques are generally referred to as of 
either "lossless" or "lossy", depending upon whether data 
is discarded in the compression process. 

A survey of conventional lossless data compression is 
provided in Simon, "Lossless Compression: How it Works", PC 
Magazine (June 29, 1993), pp. 305-13. Examples of 
conventional lossless data compression techniques include 
Huffman encoding, Pano-Shannon encoding, and arithmetic 
encoding, as well as dynamic variations of the Huffman and 
Fano-Shannon probabilistic encoding techniques. In 
general, lossless compression techniques are primarily used 
to compress entered data such as data generated from data 
processing operations, rather than sampled data 
representative of analog video or audio signals, as 
decompression will reproduce all bits of the original 
compressed data stream. 

Lossy data compression techniques, in contrast, 
provide additional data compression efficiency over 
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lossless data compression, as some amount of data is 
discarded in these techniques. As a result, lossy data 
compression techniques are generally used on sampled data, 
as some amount of inaccuracy relative to the true input 
data is tolerable; lossy data compression is, of course 
inappropriate for use on entered data streams such as those 
produced by a data processing operation. Accordingly, lossy 
data compression techniques are widely used in the field of 
compression of video and motion picture images to obtain a 
high degree of compression, as some inaccuracy may be 
tolerated. A survey of conventional lossy data compression 
techniques may be found at Simon, -How Lossy Data 
Compression Shrinks Image Piles", PC Maaa^o (July 1993) 
pp. 371 et seq. 

A popular conventional lossy data compression 
technique is referred to as the JPEG (Joint Photographic 
Experts Group) method. A description of this technique may 
be found in Barnsley and Hurd, Fractal t-^ c^.c^. 
(AK Peters, Ltd., 1993), pp. 219-228. The JPEG compression 
method initially divides the image into blocks of pixels 
and a Discrete Cosine Transform (DCT) is performed on each 
Pixel block, producing a representation of the block as 
coefficients corresponding to frequencies and amplitudes 
rather than corresponding directly to color information 
These coefficients are then quantized, or rounded off and 
a deference algorithm is performed over all quantized 
blocks in the image, in a selected scan order. This 
difference algorithm subtracts a DC term corresponding to 
the mean pixel value of a block, from the DC term of the 
preceding block. The difference coefficients are then 
scanned in a different order, such as a zig-zag order, and 
the non-zero coefficients (i.e., blocks in which a 
difference from the preceding block occurred) are coded to 
indicate the number of preceding zero coefficients (i e 
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the number of pixel blocks in which no change occurred) and 
also the value of the non-zero difference. Lossless 
compression is then often applied to the coded result to 
further compress the data. Decompression is performed by 
reversing the compression process, producing the 
displayable image. 

While the JPEG conventional video image compression 
technique is useful in obtaining high degrees of 
compression, it has been found that JPEG compression is 
incapable of being used in a real-time fashion for a motion 
picture. This is because the time generally required to 
perform the JPEG decompression of a motion picture frame 
exceeds the display time for the frame (1/30 second), and 
as a result the motion picture image cannot be decompressed 
for real-time display. Temporally accurate display of a- 
motion picture compressed according to these techniques, 
thus requires the decompression and display to be done in 
two steps, with the decompressed motion picture stored on 
video tape or another medium from which the motion picture 
can be played with the proper time base. 

Another conventional method of lossy video image 
compression referred to as Recursive Vector Quantization 
(RVQ) quantizes the pixel blocks directly, without a DCT or 
other transform, according to a set of selected reference 
tiles. £gg Simon, July 1993, op. cit. . The reference tiles 
are selected according to an iterative technique, based 
upon the accuracy of the results relative to the original 
image. As noted in the Simon article, compression 
according to the RVQ method is computationally intense and 
complex, but decompression can be done quite rapidly. 

Another type of conventional lossy video image 
compression techniques is . referred to as fractal 
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compression. As is well known in the art, a fractal is a 
mathematical image object that is self -similar, in that the 
image can be represented in terms of other pieces of the 
image. In fractal image compression, the input image is 
similarly divided into pixel groups, or tiles. Each tile 
is then approximated by a transformation (contractive, 
rotational, or both) of one or more other reference regions 
of the image. The compressed image thus consists of a full 
representation of the reference region, plus the 
transformation operators for each of the tiles. Each tile 
of the image is decompressed by performing a transformation 
of the reference region using the stored transformation 
operator for that tile. Detailed descriptions of 
conventional fractal image compression techniques and 
systems for performing the same may be found in Bamsley & 
Hurd, Fractal Image Co mpression (AK Peters, Ltd., 1993), in 
U.S. Patent No. 4,941,193, and in U.S. Patent No. 

5, 06b, 447. 



2^ Frequency and time windowin g functions 

By way of further background, the field of wavelet 
analysis has recently become popular in the analysis of the 
time and frequency response and behavior of signals. The 
following section of this application is intended to 

provide a theoretical background for wavelet analysis 
techniques in order to both convey the state of the art in 

wavelet analysis, and also to provide the necessary 

background for the person of ordinary skill in the art to 

fully appreciate the present invention. 

In the general sense, wavelet analysis is concerned 
with performing time -frequency localization of the signal 
to be analyzed (i.e,. the "input signal"). Time -frequency 
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localization refers to the analysis of a portion of the 
frequency spectrum of the input signal over a selected time 
window. As will become apparent from the description in 
this specification, time -frequency localization of an input 
5 signal enables data processing techniques to be applied to 
the signals for a wide range of purposes. 

a. Conventional analocr filtering 

10 

In the time domain, frequency-windowing is done by 
convolving a time-domain window filter function with the 
input signal; in the frequency domain, the frequency- 
windowing is done by multiplying the spectrum of the input 

15 signal with the frequency- domain transfer function of the 
filter function. Typical filter functions include low-pass 
filters (e.g., the Shannon sampling function) and band-pass 
filters. Through use of such filters, a bandwidth limited 
signal f^(t) (i.e., zero amplitude for all frequencies 

20 above a limit £2) may be decomposed into the sura of a low- 
frequency component with a series of non-zero frequency 
bands. An expression for such a decomposition is as 
follows : 

f Q(t) =f Q, Wo (t) + g Q ,i(t) +. . .+g Q/N (t) 

25 

tl] 

where is the low-pass filtered (Cd<6)0) component of 

the input signal fjj(t), where g Q ^(t) is the band-pass 
3 0 filtered signal for the ith frequency band, and where (0^ = 
fl. An ideal low-pass filtered component f Q corresponds 
to the time -domain convolution of the well-known Shannon 
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sampling function with the input signal. Ideal band-pass 
filtering may be performed by time-domain convolution of a 
filter function of the type: 

sinco^t-si na)^ t 

Tit 



[2] 



with the time-domain input signal, co n andfi/p^ being the 
upper and lower limits of the frequency band. 

Each of the low-pass and band-pass filter functions 
provide ideal frequency localization of the input signal 
fjj(t) , such that each expression and g Q ±{t ) provide 

precise information regarding the frequency' spectrum of 
input signal f fl (t) within its respective frequency band. 
However, the time localization provided by this 
decomposition is quite poor, as these filters do not 
provide very precise information about when (in the time 
domain) a signal behaves differently within certain 
frequency ranges. As many important real-world signals 
include brief periods of time of rapid transient change, 
analysis of signals decomposed in this manner will not be 
able to account for the time at which such transient 
behavior occurs. For signal analysis where time 
localization is important, conventional Fourier analysis 
techniques therefore falls short of the need. 



b. Time and frequency windowing 



Certain conventional signal analysis techniques have 
addressed this problem by time -windowing the input signal, 
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thus allowing time- localization as well as frequency 
localization. According to these techniques, a window 
function h(t) is applied to the input signal f (t) to window 
the input signal near a specified point in time t=b. This 
windowing may be considered by the integral transform: 



ffit)h(t-b) dt 



L3j 

where the bar over the function h(t-b) denotes complex 
conjugation. For a real- valued even windowing function 
h(t), this windowing process corresponds to convolution, 
allowing treatment of the windowing function h(t) as a 
filter. For example, if h(t) is the Shannon sampling 
function, the windowing process of the above equation will 
apply a low-pass filter to the input signal. However, it 
has been observed that the slow decay of the Shannon 
sampling function over time results in a very imprecise 
time-domain windowing operation. 

Those windowing functions h(t), for which the square 
of the magnitude have finite first and second moments and 
finite energy, (i.e., that decay sufficiently rapidly at 
infinity) will produce a time-window having a "center" t 
and a -radius" A h . The center t* may be calculated as I 
mean value, while the radius may be calculated as s the 
standard deviation of the windowing function around the 
mean t*. For a windowing function of radius A h , the width 
of the time -window will be 2A h , commonly referred to as the 
RMS duration of the windowing function h(t). in the 
frequency domain, if the square of the magnitude of the 
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Fourier transfer* h(«) of the windowing function h(t) has 
fimt. first and second moments, the frequency domain 
windowing function fi(«) wiU ^ a center ^ ^ g width 

2Afr calculated in a manner similar to the mean and 
standard deviation (doubled) of the frequency domain 
function the width 2A fi is usually called the RMS 

bandwidth of the windowing function h(t) . If the RMS 
duration A h is f±nlte , then the time-domain windowing 
function h(t) is a time-window; similarly, if the rms 
bandwidth 2A h is finite, the frequency-domain windowing 
function f]{b>) is a frequency window. 

Referring back to the ideal low-pass and band-pass 
filter time-domain functions noted above, it will be 
readily apparent that their first moment is infinite 
meaning that neither of these ideal filter functions can 
serve as time windows if used as windowing functions h(t) . 
However, the frequency domain representations of the ideal 
low-pass and band-pass filter functions have finite RMS 
bandwidth aAfi. so that these filters provide ideal 

frequency- localization as is evident from their ideal 
nature . 



As discussed above, accurate analysis of real-world 
signals containing transient behavior requires both time- 
localization and frequency-localization. The Uncertainty 
Principle has identified those windowing functions h that 
provide both time windowing and frequency windowing as 
those functions that satisfy the following inequality 



A h A h * * 

[5] 
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It has also been previously found that the only types of 
windowing functions h(t) that achieve the lower bound of 
the Uncertainty Principle are those of the. form: 

h{t) =ce J ' at e (t - 2>)2 /4o 

[6] 



for some constants a, b, c and a with a > 0 and c * 



0, 



Further indication of the presence of a time -frequency 
window by any windowing function h(t) may be obtained 
through the Parseval identity. The generalized windowing 
function h(t) noted above corresponds to: 

ao 

CD 

I fit) hit-h) dt~-±- 1 f (©) e^STofdo 



[7] 



With reference to the left side of equation [6] , the time 
window is given by: 



[b + t" - A„, b + t* + Aj 



[8] 



such that the time window that is centered on t = t* is 
shifted by the parameter b; the time window also has a 
radius of A h in the time dimension. Similarly, with 
reference to the right-hand side of equation [s] , the 
frequency window is given by: 



[9] 
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and is thus centered at A/=6> 4 „ it h width 2Afi. Figure la 
illustrates the location of time -frequency window 2 of 
equation [6] in a time -frequency coordinate system. 

For causal real-valued window functions, where: 

=£(«)" 

the function \f\(cu)\ is an even function, so that the center 

is located at a=o and the frequency window of equation 
[S] becomes 

l " A h' A tf [io] 

However, while the time -frequency window of the filtering 
function h(t) may be moved along the time axis of Figure la 
by . changing the value of b, the window is fixed in 
frequency at the center frequency or, in the case of 
real-valued even functions as noted above relative to 
equation [10], is fixed at a center frequency CO, = 0 as 
shown in Figure lb. This fixation in frequency of ' the 
time-frequency window in limits the usefulness of the 
windowing process of equation [3] . 



— Th e Short-Tims *Vy^-iq r Tr^.fn^ (STE-T) 

Rudimentary Fourier theory indicates that translation 
in the time-domain corresponds to a phase-shift in the 
frequency domain, and conversely that a phase-shift in the 
time-domain corresponds to a translation in the frequency 
domain. Accordingly, a phase shift in the windowing 
process of equation [3] should allow for sliding of the 
frequency window along the frequency- axis . 



SUEHfOIESfiEErflffllE^ 



WO 95/19683 



PCT/US95/00563 



12 

Considering a real -valued window function 0(t) which 
serves as a low-pass filter (i.e., 0(fl=O) = 1) , and for 
which 0(t), |t| M 0(t) and t<p(t) are in L 2 , the short-time 
Fourier transform of (pit) is defined by: 

00 

(G^f) (b, S ) : =| f ( t) e-^ c <|> ( t-b) dt 

— OM 

[11] 

for f 6 L 2 . The short-time Fourier transform (STFT) of 
equation [11] is also referred to in the literature as the 
Gabor transform. Applying the Parseval identity results in 
the following expression for the STFT, from which the 
center and radii of the time and frequency windows are 
apparent : 

00 

(G^f) (b. I) f f (») -5 )d» 

7U J 

— oo 

[12] 

The short-time Fourier transform of equations [11] and 
[12] provide the improvement over the simple time -windowing 
process of equation [3] in that the frequency window 
function <p(U) is able to slide along the frequency axis. 
The frequency window for a windowing function <j>((t)) that 
otherwise (i.e., for £=0) has its center 10+ at ft/=0, is now 
localized to frequencies near &) » £ : 

a;] [13] 



WO 95/19683 



PCT/CS95/00563 



13 



Similarly, where the center t* of the windowing function 
(pit) is also otherwise (b=0) located at the origin, the 
time window is now localized near t=b as follows: 

[b - A„ b + A,J [14] 

The STFT thus allows for sliding of the time- frequency 
localization windows both in time and in frequency, merely 
by changing the values of the phase-shift factors b, £, 
respectively. Figure lc illustrates the position in time- 
frequency space of two time -frequency windows 5 Q , 5 
having phase-shift factors (b Q , $ Q ) , < bi> 
respectively. As a result, the STFT allows the lowpass 
window function to perform bandpass filtering by changing 
the value of However, as is evident in Figure lc, the 

widths of the windows are invariant with changes in time- 
shift factor b or frequency- shift factor $ . Accordingly, 
while analysis is improved through use of the short-time 
Fourier transform, inaccuracies due to undersampling have 
been observed for those transient periods of time in which 
rapid changes (i.e., amplitudes over a wide range of 
frequencies) exist. Accordingly, not only is the ability 
to slide the localization windows in both time and 
frequency desired, but it is also desirable to allow for 
scaling of the window widths as a function of time or 
frequency. 
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3 . Wavelet analysis 



a . Theory 

Wavelet analysis techniques address the need for time- 
frequency localization windows of flexible size, primarily 
by introducing a scale parameter that is mapped to the 
frequency variable. As a result, the scale parameter will 
change the widths of the windows as a function of 
frequency/ thus changing the aspect ratio of the windows 
with changes in frequency. Since the Uncertainty Principle 
requires the area of the windows to remain above a certain 
value, the time window width decreases and the frequency 
window width increases proportionally as the center 
frequency £ increases. The narrowing of the time window 
and widening of the frequency window for high frequency 
environments more precisely detects and analyzes these high 
frequency portions of input signals. 

The basic wavelet transform is known as the integral 
wavelet transform, or IWT. The IWT includes a scale 
parameter a- in its definition as follows : 



[15] 



As such, the window function If/ narrows with changing values 
of the scale parameter a, such that the time width of Ifl 
decreases with decreasing a. The windowing function ifl (t) 
used in the IWT of equation [15] is to be real-valued as 
before, but the IWT constraints also require Ifl (t) to be a 
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bandpass filter rather than a low pass filter, such that 
its Fourier transform *ft,. 0 ) = 0, stopping at least zero 
frequency components of the signal. Since the windowing 
function ^(t) is real-valued, its Fourier transform 
satisfies : rv ' 



[16] 



so that |p (« ) | is an even function. Because only 
nonnegative frequencies are of interest, . and since *( t ) is 
a band-pass filter, the Fourier transform j (ft, ) need only 
be considered as a frequency window in the frequency domain 
[0, ») , with the centers and widths of the frequency window 
function f («) being modified as a refiult ^ a window . ng 

function If, (t) in! 2 , for which |t|*( t ) and * (t) are also 
in L such that ft> (t) is real-valued, and where j (ft, =0 ) - o 
the one-sided (i.e., nonnegative) frequency window center 
» + as a function on the domain [o, « } is defined as: 



0) + : = 



. = o 



/!♦(©) | 2 dG> 



[17] 



and the one-sided radius of lp{id) ± B defined 



as: 
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AJ: = 



\ 



j (G)-G>:) 2 |<|f (G)) | 2 d0) 

0 _ 

QO 

/|$(o>) | 2 dO) 



[18] 



This allows the generation of an integral wavelet 
transform (IWT) using a normalization factor a"* based upon 
the scale parameter a, which scales the time-width of the 
window as a function of frequency. For a windowing 
function $ (t) that satisfies the conditions for equations 
[17] and [18] above, the IWT is defined as follows: 



yfal * 



[19] 

For the IWT of equation [19] , the bandpass window-function 
<Mt) is commonly referred to as the analyzing wavelet. 

As is known in the wavelet analysis field, and given 
the foregoing discussion, it is important that the integral 
wavelet transform W^ allows for frequency localization 
where the width of the time window is mapped to the 
frequency domain, and in which the frequency window can 
slide along the frequency axis. For finite -energy real- 
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valued input signals f(t), and since 'f (t) is real, the 
following relationship holds: 

f ( - w ) e^tjr (-ao» =jf(o>) e-^ u ^mST 

[20] 

Through the Parseval identity, one can then derive the IWT 
as follows : 



w 

(«r,f) .(2), a) -&Ref£ <») e^r, (a (o>--^ ) ) do, 

0 3 

[21] 

for all f £ L 2 R , where Ui + * is the one-sided center of 0 (a; ) 
on the domain [0, » } , ^ where ^ ±B defined ag fdllQwag 



n (o>) :=$(o)+cot) 



[22] 



As noted above, it is desirable to map the scale 
parameter a to the frequency at which the time-frequency 
window is to be localized. Accordingly, the scale 
parameter a is mapped to the shift frequency parameter £ as 
follows : 
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a 



for some c > 0 f where c is a calibration constant. 
Substituting for the scale parameter a defines the IWT as 
follows : 



(*£f) (2>,£) : = (W t f) (b f a)=(W }jf f) (jb, -|) 

[24] 

It is convenient to set c=d) + * , so that: 

(Kf) ib. S) =^§Ref£ (o>) (a (o-$) ) do 



[25] 

This produces a frequency window i;(a«tl^)) that slides 
along the frequency axis with the value of having a 



range; 



[26] 
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The width of this window thus increases at higher 
frequencies £ , as reflected in smaller values of the scale 
parameter a. In terms of the scale parameter a, the width 
of the frequency- window is as follows: 



[27] 



where the frequency shifting term { -«//«, such that the 
frequency-width of the frequency window increases with 
increasing frequency { (decreasing values of a) . Along the 
time-axis, the time window of the IWT of equation [19] is 
given by: 



[28] 



As a result, the width of this time-window is 2a^, which 
decreases at higher frequencies £ (and lower values of a) , 
and which increases at lower frequencies For the 

transform of equation [15] , Figure id illustrates three 
time-frequency windows 6 Q , 6^ 6 2 , with varying translation 
factor pairs (b Q , * 0 >, (b^ ^) , (b 2 , ^) , respectively. 
As is evident in Figure Id, both the time-width and 
frequency-width of windows 6 vary with varying shift 
frequency such that the time-width decreases and the 
frequency-width increases with increasing £ . 
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Accordingly, it should now be apparent to those in the 
art that wavelet analysis is capable of providing increased 
accuracy analysis of signals, particularly those including 
transient components, considering that, for higher 
frequency analysis, the width of the time window decreases 
and the width of the frequency window increases. This 
ensures adequate sampling of the input signal function, and 
also allows for determination of the exact time in the 
signal at which the transient event occurred. 

As is known in the art of wavelet analysis, however, 
the definition of the proper wavelet function Ifl (t) is of 
great importance. Various specified analyzing functions 
have been used in wavelet analysis, with the selection of 
the function made according to computability, or according 
to attributes of the signal under analysis. 

k» Prior wavalPt^p 

Wavelet signal analysis has been applied to signals 
produced in seismic exploration for oil and gas, as 
described in U.S. Patent No. 4,599,567. This reference 
describes a wavelet analysis technique using Morlet's 
wavelet as the analyzing wavelet. Morlet's wavelet is a 
sinusoid limited by a Gaussian probability envelope to a 
finite duration; the envelope may or may not be skewed 
toward either the leading or lagging edge of the time- 
domain envelope of the wavelet, as desired for the type of 
signal under analysis. This reference also discloses 
circuitry for performing the wavelet analysis of an 
incoming signal using such analyzing wavelet. 

Another finite duration analyzing wavelet was proposed 
by Yves Meyer. This analyzing wavelet is substantially a 
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finite duration uniform magnitude level over the wavelet 
window, analogous to the Shannon sampling function. 

Other wavelets have been proposed which are not 
expressible by a mathematical formula, but instead are 
utilized as numeric wavelets. These wavelets include the 
Battle-Lemanne wavelet, which is a spline function of 
infinite duration, and the Daubechies wavelet, which is a 
fractal wavelet of finite duration. The lack of explicit 
formulae for these wavelets limit their applicability for 
rapid and accurate computer implementation of the wavelet 
analysis technique in computer hardware. 

Another previously published wavelet is the Chui-Wang 
wavelet, which is a wavelet of finite duration but which 
may be expressed in an explicit formula. 

The support for each of the prior wavelets noted above 
is over an unbounded interval. However, since real-world 
problems require the application of the wavelets to bounded 
intervals, wavelet analysis of input signals using these 
prior wavelets result in errors of the type commonly 
referred to as "boundary effects". 

Figure 2 graphically illustrates the reason for 
boundary effects arising from such conventional wavelets 
Conventional wavelet 7 illustrated in Figure 2a is first - 
order spline-wavelet that is based on a function ? having 
moments with the following properties (for i - o l 
1/ and some m a l) : 



WW 

/ 
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where a, b are scaling parameters as discussed above. The 
conventional wavelet 7 of Figure 2a is not orthogonal over 
a bounded interval [c, d] , however, meaning that: 



/ 



[30] 

for all integers i > o. 

Figure 2 graphically illustrates the performance of 
the IWT using wavelet 7 having the above noted properties 
at a point in the time series which happens to be at or 
near the boundary of an interval [c, d] . Data points f 



f 



3 / f 4 correspond to input signal samples within the 
interval, which are plotted against wavelet 7 in Figure 2; 
in this example, for purposes of explanation, the input 
signal sample data closely matches the shape of wavelet 7 
within the interval [c. d] . The position of wavelet 7 
corresponds to the position, in performing the IWT, of the 
sample point of interest at the boundary value t=a. Since 
wavelet 7 at this position requires support outside of the 
interval [c, d] for which input signal data exists, the 
zero values that must be assumed for the data points f , f 
outside of interval [c, d] necessarily fail to match 
wavelet 7. This will result in an non-zero result for the 
IWT, even where the input data signal within the interval 
[c, d] exactly matches wavelet 7. As is well known in the 
field of signal processing, this inaccuracy due to wavelet 
7 requiring support outside of the bounded interval [c, d] 
is made manifest by boundary effects at the edges of the 
bounded interval, since the unbounded wavelet 7 fails to 
accurately represent the series of actual input signal 
sample data. 
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In the field of video image compression and analysis, 
boundary effects greatly affects the quality of the image 
displayed after compression and decompression. This is 
because the boundary effects will appear as false image 
data at the edges of pixel blocks corresponding to bounded 
intervals, and also at the edges of the image (even if no 
subdivision of the image is performed) . The inaccuracies 
due to boundary effects also limit the ability to magnify 
an image when displayed, considering that the magnification 
will make the boundary effect errors to become even more 
visible. 



c . Th e boundarv-spline-wavelet: 

By way of further background, a bounded interval 
wavelet is described in Chui and Quak, "Wavelets on a 
Bounded Interval", Numerical Methods of ATmmvin.aHnn 
Theory, Volume 9 (Dec. 1992), pp. 53-75, incorporated 
herein by this reference. This wavelet, which has an 
explicit formula, is not only a function of the time 
variable t, but is also a function of the position of the 
sample within the interval [c, d] , so as to account for 
boundary effects. In effect, sample locations near the 
boundaries of the interval will correspond to different 
wavelet shapes than will sample locations within the 
interval that are away from the boundaries. Boundary 
effects are eliminated as the boundary wavelets do not 
require support outside of the interval. 

Referring now to Figures 3a through 3d, an exemplary 
set of four first-order wavelets 8 according to the Chui- 
Quak boundary- spline -wavelet approach are illustrated. 
Figure 3a illustrates the shape of "boundary" wavelet 8 
for a sample location near the boundary t=a of the interval 
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[a, b] , while Figure 3d illustrates the shape of boundary 
wavelet 8 b for a sample location near the boundary t=b of 
the interval [a, b] . Figures 3b and 3c each illustrate the 
shape of -inner" wavelets 8. for sample locations within 
the interval [a, b] away from the boundaries. As is 
evident from Figures 3a through 3d, boundary wavelets 8 
8 b , have different shapes than inner wavelets 8. (which 
have the same shape as one other). As is further 1 evident 
from Figures 3a through 3d, neither inner wavelets 8. nor 
boundary wavelets 8 a , 8 b require support outside of the 
interval [a, b] , or: 



b 

/ 



Gt 



t*i|r(-£-£)cte-Q 

3 



[31] 



for i = 0, 1, . . . , m-l, and for some m a 0, for the 
entire set of wavelets 8 of Figures 3a through 3d. 
Accordingly, application of the set of boundary-spline- 
wavelets 8 to actual real-world data, for which the. time 
interval is necessarily bounded, will not produce boundary 
effect artifacts. 

Other boundary-wavelets are known, as described in 
Daubechies, "Two recent results on wavelets : Wavelet bases 
for the interval and biorthogonal wavelet diagonalizing the 
derivative operator". Recent Adv ances in w ave let A^l y ^ a 
Schumaker and Webb, ed. (Academic Press, 1993), pp. 237-58 '. 
These wavelets are not spline functions, and do not have 
explicit formulae. As a result, it is believed that these 
wavelets are of limited effectiveness in video image 
compression and decompression. 
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±j Objects of the invention 

It is therefore an object of the present invention to 
apply wavelet analysis tools to the task of video image 
compression for storage and transmission. 

It is a further object of the present invention to 
provide an apparatus for performing video image compression 
according to a selected wavelet. 

It is a further object of the present invention to 
provide an apparatus for receiving a compressed video image 
and to decompress the same for real-time playback of the 
stored or communicated video image information. 

It is a further object of the present invention to 
provide such a method of compression and decompression such 
that the compressed image may be played locally at real- 
time . 

It is a further object of the present invention to 
provide such a system and method which allows for high 
accuracy magnification of the decompressed image, with much 
reduced incidence of edge effects. 

It is a further object of the present invention to 
provide such a system and method which provides a high 
degree of compression. 

It is a further object of the present invention to 
provide such a system and method which can utilize dynamic 
compression on a frame-by- frame basis, such that high 
frequency frames may be compressed to different ratios than 
low frequency frames. 
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It is a further object of the present invention to 
provide such a system and method which facilitates 
interactive display of a motion image, including insertion, 
editing and repetitive display. 

It is a further object of the present invention to 
provide such a system and method which provides the ability 
for slow display systems to skip certain frames so that a 
motion picture can be displayed on a real-time basis by 
such slower systems, although with fewer frames per second. 



It is a further object of the present invention to 
provide such a method and system which allows for division 
of an image into several portions for purposes of 
compression and communication or storage, with subsequent 
display of the full image after decompression. 

It is a further object . of the present invention to 
provide such a method and system which allows for higher 
lossy compression ratios by further quantization, as useful 
in compressing and decompressing high-quality 24 -bit still 
images . 



Other objects and advantages will be apparent to those 
of ordinary skill in the art having reference to the 
following specification, together with its drawings. 
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Summary of th e Invent- -ii-m 

The present invention may be implemented into a method 
and apparatus for compressing and decompressing video 
images. The compression/decompression system receives 
video image data in a suitable format, such as eight -bit 
Portable Grey Level format, and includes circuitry for 
decomposing each component of the image into low and high 
frequency components by way of boundary-spline-wavelet 
decomposition. The system also includes circuitry, for 
quantizing the high frequency portions of the 
decomposition. The decomposition may be repeated until the 
desired compression ratio is obtained. Lossless 
compression may then be performed on the compressed image 
following which the compressed data may be stored for 
archival purposes, or may be transmitted to a receiving 
station. 

According to the present invention, in decompression 
mode, the system performs lossless decompression of the 
stored or communicated image data, according to the 
compression scheme utilized. Reconstruction of the image 
is then performed by way of the boundary- spline -wavelet 
approach, in a reverse fashion relative to the image 
compression. The result of the decompression is then 
formatted as desired for display. 
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Brief D escription of the Drawings 



Figures la through Id are time -frequency plots 
illustrating the concept of time -frequency windowing, as 
described in the background of the invention. 

Figure 2 is a graphical illustration of a wavelet with 
non-zero moments of any order relative to the interval [c, 
d], as applied to an input signal, and which is the source 
of boundary effects. 



Figures 3a through 3d are graphical illustrations of 
an exemplary set of boundary-spline-wavelets as used in the 
15 preferred embodiment of the invention. 

Figure 4a is an electrical diagram, in block form, of 
a system for producing and commiinicating or storing 
compressed video image information according to the 
20 preferred embodiment of the invention. 

Figure 4b is an electrical diagram, in block form, of 
the image format converter in the system of Figure 4a. 
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Figure 5 is an electrical diagram, in block form, of 
a compressor according to the preferred embodiment of the 
invention . 

Figure 6 is a flow diagram illustrating a method of 
compressing video image data according to the preferred 
embodiment of the invention. 

Figure 7 is a detailed flow diagram illustrating a 
method of performing the boundary- spline -wavelet 
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decompression according to the preferred embodiment of the 
invention . 

Figure 8 is a series of frequency domain plots 
illustrating the decomposition of the video image according 
to the preferred embodiment of the invention. 

Figures 9a through 9c are matrices illustrating the 
effect of decomposition of the video image according to the 
preferred embodiment of the invention. 

Figure 10 is a flow diagram of a preferred matrix 
algebra technique for performing the boundary- spline - 
wavelet decomposition of Figure 7. 

Figure ll is a chart illustrating the relationship of 
the matrices in the process of Figure 10. 

Figures 12a through I2e illustrate an example of a 
video image decomposed according to the preferred 
embodiment of the invention. 



Figures 13a through 13c illustrate the quantization of 
the decomposed video image of Figures 12a through I2e 
according to the preferred embodiment of the invention. 

Figure 14 is a diagram of the construction of a 
compressed video image frame for transmission according to 
the preferred embodiment of the invention. 

Figure 15a is an electrical diagram, in block. form, of 
a decompressor according to the preferred embodiment of the 
invention . 



WO 95/19683 



PCT/US95/00563 



30 



Figure 15b is an electrical diagram, in block form, of 
a format converter useful in the decompressor of Figure 



15a 



Figure 16 is a flow chart of a process for 
decompressing video image data according to the preferred 
embodiment of the invention. 

Figure 17 is a detailed flow chart of a process for 
reconstructing video image data that were decomposed 
according to the boundary-spline-wavelet approach of the 
preferred embodiment of the invention. 

Figure 18 is a chart illustrating the relationship of 
the matrices as used in a preferred matrix algebra routine 
for the reconstruction of Figure 17. 

Figure 19 is an electrical diagram, in block form, of 
a decompressor system according to an alternative 
embodiment of the invention. 
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Detailed Description of the P^f g - r red EmhnH^o^ 
— Video image data communication Ry afpni 

Referring now to Figure 4a, the construction of a 
video communication system 8 according to the preferred 
embodiment of the invention will now be described in 
detail. m this embodiment, system 8 is specifically 
designed for communicating compressed motion picture data; 
it is of course contemplated that system 8 may 
alternatively be used to communicate compressed data 
representative of a series of still video images. m 
addition, as will be described in further detail 
herexnbelow, system 8 may alternatively or also be used for 
archival storage of motion picture or still video image 
data, and the subsequent retrieval and display thereof. 

As shown in Figure 4a, in the context of video data 
communication, the transmission end of system 8 includes 
vadeo source 12 and compression system 10, while the 
receiving end of system 8 includes decompression system 20 
and video display 26. As illustrated in Figure 4a, video 
source 12 may include digital video source I2d which may be 
a conventional device such as a CD-ROM drive, scanner 
digital electronic network connection, or similar unit, or 
alternatively may be a computer storage unit such as a disk 
that contains digital video information. Video source 12 
may also or instead include analog video source 12a, which 
may be a video camera, VCR unit, television broadcast or 
cable receiver, or another conventional source of analog 
video information. m any case, video source 12 provides 
digital or analog signals indicative of the images to be 
communicated or stored by the system of Figure 4a. 
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Communication network system 15 is a conventional 
analog transmission or electronic digital communications 
network, or both an analog and digital network when 
including the appropriate clusters, analog- to-digital and 
digital-to-analog converters, and other necessary 
apparatus. Network 15 may be realized according to any 
conventional technology, including hard- wired cable, fiber 
optic cable, broadcast or satellite transmission, and the 
like. It is further contemplated that network 15 may be 
implemented merely by the physical transportation of 
portable media 22' such as floppy diskettes, CD-ROMs and 
the like. Regardless of the implementation, network 15 is 
connected between compression system 10 and the input of 
decompression system 20 to communicate compressed video 
image data therebetween. 



It is specifically contemplated that the video data 
transmission effected from compression system 10 may be of 
the broadcast type, such that a single transmission end 
(source 12 plus compression system 10) may communicate 
simultaneously or in sequence to multiple receiving ends 
(decompression system 20 and display 26). For example, 
compression system 10 may be located at a television or 
movie studio, or at a local cable television system "head- 
end", with multiple receiving ends (decompression system 20 
and display 26) located at homes, offices, or local 
theaters, depending upon the particular transmission being 
effected. 

Figure 4a also illustrates the optional use of system 
8 in the context of the archival storage of motion picture 
or still video image data, and its later retrieval and 
display thereof. Disk storage unit 22 is coupled to 
receive the compressed video data from compression system 
10 for storage. An example of disk storage unit 22 is a 
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large disk (e.g., having capacity on the order of 1000 
Gigabytes) / alternatively, or in addition to the large disk 
unit, disk storage unit 22 may be associated with a 
mainframe or supercomputer . to provide services such as on- 
line collection of video images into a library form. Disk 
storage unit is also coupled to decompression system 20, 
which is operable in this context to receive compressed 
video data therefrom for display on video display 26. The 
subsequent retrieval of the archived information in disk 8 
may be made via a digital communications network (such as 
network 15 described hereinabove) , or alternatively by way 
of a portable data storage medium such as a tape or 
diskette . 

Compression system 10 includes data format converter 
14 that is functionally connected between video source 12 
and compressor/coder 16. Format converter 14 is of 
conventional construction as used to convert the format of 
the video image data from video source 12 into the format 
suitable for compression by compressor 16. In the example 
described hereinbelow, the format utilized by compressor 16 
is eight-bit Portable Grey Level (PGM) , although fewer than 
eight bits may be used in the PGM format, depending upon 
the available hardware devices and architecture. For color 
images, the format is RGB -PGM, where each of the red, green 
and blue color components is separately stored and 
communicated in PGM format. The PGM image format will be 
described in further detail hereinbelow. Accordingly, 
format converter 14 converts the image data from video 
formats such as PCX, IMG, GIF, TIF, RLE, NTSC, PAL, and the 
like into PGM or RGB -PGM format. Of course, if the output 
from video source 12 is already in PGM or RGB -PGM format, 
format converter 14 is unnecessary, and compressor 16 may 
directly receive the data from video source 12. 
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Referring now to Figure 4b , the construction of format 
converter 14 according to the preferred embodiment of the 
invention will be described in detail, for the example 
where the output provided by format converter 14 is in RGB- 
PGM format. As noted above, format converter 14 can 
receive video data from either analog or- digital video 
sources 12a, I2d, respectively, and as such two input ports 
are provided to format converter 14 along with two 
functional devices to reformat the received data into the 
desired format. 

In the example of Figure 4b, digital video image data 
presented by digital video source 12d is received by format 
decoder and color expansion device 21. The format decoder 
portion of device 21 decodes the data from the format 
presented by source 12d, such format being PCX, IMG, TIF, 
GIF, RLE, YUV, etc., into a color matrix, or color palette 
table, representation of the image signal. Format decoder 
circuitry is well known in the art, such as described in 
Rimmer ' Supercharged Bitmanr^H r.~, r v,^ (Windcrest/McGraw 
Hill). This decoding is followed by a color expansion 
operation performed by device 21, such that the output 
therefrom is in raw RGB format, with each pixel represented 
by successive bytes of red, green and blue color 
components. Conventional circuitry for performing such 
color expansion is readily available in the field, 
including such devices as color palette RAMs . The output 
of format decoder and color expansion device 21 is 
presented to color separator 25. 

On the analog side, analog video information is 
presented by analog video source 12a to analog screen 
grabber and digitizer device 23 . Device 23 first converts 
the analog input data from its NTSC or PAL format, by the 
screen grabber capturing the input screen data. Device 23 
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then digitizes each pixel of the captured screen, in the 
conventional manner. Conventional circuitry for performing 
the screen grabbing and digitizing functions may be used to 
implement device 23. The output of device 23 is also in 
the raw digital, or binary, RGB format as presented by 
device 21. 

Color separator 25 is a conventional digital filter 
for separating the raw binary RGB signal presented by 
either of devices 21, 23 into RGB - PGM format. As is well 
known in the art, the RGB -PGM video data format includes, 
for each pixel, three eight-bit components. These three 
eight -bit components correspond to the intensity of the 
red, blue and green colors, respectively, to be displayed 
for that pixel. Of course, other byte widths may be used 
to digitally represent the intensity of each color 
component . 

The RGB -PGM format used in this preferred embodiment 
of the invention decomposes the input video image into 
three screen representations, one for each of the red, blue 
and green color components. Each pixel location in the 
input image is thus represented by a pixel value in each of 
the red, blue and green decompositions, indicating the 
intensity of that color component at that pixel location of 
the image. In the eight -bit PGM format, the intensity of 
each color component for each pixel can thus range from 0 
(black, or no intensity) to 255 (full intensity) . The 
eight-bit PGM format' is particularly convenient v for 
processing by microprocessors or computers using ASCII 
coding and programming environments, as the fundamental 
storage unit is the byte in these environments. 

The three channels of RGB- PGM data produced by color 
separator 25 is then presented to compressor 16, either 
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sequentially or in parallel, for compression as will be 
described hereinbelow. According to this embodiment of the 
invention, each of the R, G, B components are compressed, 
decompressed, and otherwise processed separately from the 
other components for that image. 

Of course, if the data presented by video source 12 
specifically digital video source l2d, is a i ready in RGB '_ 
PGM format, format converter 14 is not necessary in the 
system of Figure 4a. 

Compressor 16 includes the necessary compression 
circuitry, an example of which is described in detail 
hereinbelow, for compressing the formatted digital data 
according to the boundary-spline-wavelet technique that is 
also described in detail hereinbelow. compressor 16 may 
also mclude coding circuitry for formatting the compressed 
data into the suitable format for communication over 
network 15 (or storage in disk 22) . *s will also be 
described in further detail hereinbelow, specific 
information regarding attributes of the communicated or 
stored information may be specifically included in the data 
coded by compressor 16. 

Similarly, decompression system 20 includes 
decompressor 18 coupled to network 15 (or to disk 22 as 
the case may be) . Decompressor 18 receives the transmitted 
or stored image data, reformats it into a form suitable for 
decompression, if necessary, and decompresses the data 
Decompressor 18 in this embodiment of the invention 
communicates the decompressed image data to format 
converter 24, which converts the decompressed data to the 
appropriate format for display by display 26. 
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As illustrated in Figure 4a, display 26 may be 
implemented as a digital display 26d, to which digital data 
may be directly applied thereto; alternatively, display 26 
may be implemented as a conventional analog display, with 
the appropriate NTSC or other analog video format data 
applied thereto. According to this example, the output 
data from decompressor 18 is in PGM or RGB -PGM format, and 
thus format converter .24 will convert the PGM data into 
PCX, IMG, GIF, TIF, RLE, NTSC, PAL, RGB, or another display 
format. Of course, if digital display 26d is used and is 
capable of receiving and directly displaying PGM format 
data, format converter 24 is unnecessary. 

■2-! Constru ction of the compressor 

Referring now to Figure 5, the construction of 
compressor 16 according to the preferred embodiment of the 
invention will now be described in detail. It is, of 
course, contemplated that other architectural arrangeit^nts 
of circuitry may be used in the compression of video data 
according to the present invention. Specifically, it is 
contemplated that a conventional general purpose computer 
system may be capable or. performing the compression of the 
video image data according to the present invention. 
However, the example of Figure 5 incorporates a preferred 
embodiment of the circuitry for performing the data 
compression functions to be described herein. 

In this example, compressor 16 preferably includes one 
main RGB -PGM channel controller 28 and three substantially 
identical channel compressor subsystems 29 R , 29 , 29 
Three channel compressor subsystems 29 R , 29 29 are 
provided according to this embodiment of the invention, 
considering the separation of the input video data into the 
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three component channels by format converter 14 described 
hereinabove. Compressor 16 also preferably includes data 
flow interface 39, for receiving data trom each ot channel 
compressor subsystems 29 and presenting the same to network 
5 15. 

Main controller 28 receives the three channels of rgb- 
PGM data from format converter 14, and forwards each 
channel of data separately to the appropriate channel 
compressor subsystem 29 Rf 29 Q , 29g. Alternatively, 
compressor 16 may be implemented to have only a single 
channel compressor subsystems 29 which processes each 
channel of data sequentially, in which case main controller 
28 would control the sequential forwarding of image data to 
the single compressor subsystem 29. 

Main controller 28 is connected by way of the 
appropriate bidirectional buses and ' control lines to 
control the functions within compressor 16. In addition to 
its control of subsystems 29, main controller 28 also 
controls the timing, feedback and sending operation of 
compressor 16, including control of the data flow interface 
39. As such, main controller 28 is preferably a general 
purpose programmable microprocessor or other central 
processing unit (CPU) of sufficient computational power and 
capacity to process some or all of the image data and to 
control the performing of the image compression functions 
to be described hereinbelow. It is contemplated that 
microprocessors having performance levels similar to, or 
30 greater than those of the 80486 type (including those 
available from Intel Corporation or Cyrix Corporation) , of 
the 68 040 type (including those available from Motorola) , 
and of the SPARC processor type (available from Texas 
Instruments Incorporated or Sun Microsystems, inc.) will be 
35 suitable for use as main controller 28 in compressor 16. 
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The construction of channel compressor subsystems 29 

will now be described in detail relative to the 

construction of subsystem 29 R as shown in Figure 5; it is 

contemplated that the other subsystems 29„, 29^, will be 

* ■ KG* 
similarly constructed. Each channel compressor subsystem 

29 according to this embodiment of the invention is 

specifically designed to perform the functions of boundary 

spline wavelet decomposition, quantization and lossless 

compression, as used in the compression operation according 

to this embodiment of the invention. 

Channel compressor subsystem 29 according to this 
embodiment of the invention includes the circuit functions 
of digital matrix process 30, timing circuit 37, 
quantization processor 32, lossless compressor 34, program 
data memory 35, and multiple memory banks 36 for storage of 
image data. 

Timing circuit 37 performs the functions of receiving 
the PGM format channel data from main controller 28 and 
forwarding the received data to memory banks 36 in its 
subsystem 29. In addition, timing circuit 37 controls the 
timing and feedback among the other functional components 
of channel compressor subsystem 29, including the matrix 
operations performed by digital matrix processor 30, 
quantization performed by quantization processor 32, 
lossless compression performed by lossless compressor 34, 
and accesses to memory 35, 36. 

Digital matrix processor 30 is a processing circuit of 
conventional architecture that is specifically suitable for 
performing vector and matrix operations as used in the 
decomposition of image data according to the preferred 
embodiment of the invention, as will be described in detail 
hereinbelow. As will be apparent from the description 
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below, these operations include the retrieval of pre- 
calculated matrices from program data memory 35, the matrix 
operations utilized in performing boundary-spline-wavelet 
decomposition of the video channel data, and storage of the 
results in memory banks 35. Examples of presently 
available components suitable for use as digital matrix 
processor 30 include digital signal processors such as the 
i860 processor available from Intel Corporation and the 
TMSC40 digital signal processor available from Texas 
Instruments Incorporated, and also general purpose 
microprocessors such as those of the 80386 and 80486 
families available from Intel Corporation, and of the 68030 
and 68040 families available from Motorola. 

Quantization processor 32 is a logic circuit for 
filtering the data corresponding to decomposed images in 
order to achieve the desired compression ratio. it is 
contemplated that conventional processing circuitry or 
custom logic circuitry for performing this function in the 
manner described hereinbelow, will be readily apparent to 
one of ordinary skill in the art. According to this 
preferred embodiment of the invention, quantization 
processor 32 may be selectably controlled to perform the 
quantization process according to various selectable modes 
These modes are selectable by way of quantization mode 
register 31, which stores a three digit code defining the 
type of quantization to be performed by quantization 
processor 32. An example of the codes storable in 
quantization mode register 31 and their corresponding 
quantization modes are as follows: 
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No quantization 
thresholding 
scalar quantization 

JPEG quantization (i.e., using tables) 
reserved for other quantization modes 
(e.g., vector quantization) 



Lossless compressor 34 may be implemented by way of a 
conventional digital signal processor such as the TMSC40, 
programmed in such a manner as to perform lossless 
decompression upon the results of the quantized output from 
quantization processor 32. The lossless decompression 
performed by lossless decompressor 34 is according to the 
desired conventional technique, such as Huffman encoding, 
adaptive Huffman encoding, arithmetic encoding, LSQ coding, 
and the like. Alternatively, lossless compressor 34 may be 
implemented as a custom logic circuit for providing this 
function. The output of lossless compressor 34 is 
preferably compressed data for the channel (R, g, B) in 
bitstream format, for application to data flow interface 



Data flow interface 39 provides an interface between 
compressor 16 and network 15, and as such must gather the 
bitstream output from lossless compressors 34 in each of 
the subsystems 29 and arrange the same into a suitable 
format for transmission. Data flow interface 39 also 
provides a feedback signal to main controller 28 upon 
transmission of a frame of compressed data, based upon 
Which main controller 28 may commence the processing of the 
next image frame to be compressed. 

The preferred format in which, the output compressed 
data from data flow interface 39 is communicated over 
network IS will be described in further detail hereinbelow. 
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This example of compressor 16 is intended to support 
the compression of high definition real-time true color 
video image data, where "true color" indicates the use of 
twenty- four bits of color information for each pixel, 
resulting in 16.7 million possible colors. The frame rate 
for compressor 16 is intended to be on the order of thirty 
frames per second, so as to support "real-time" video image 
compression. 

As noted above, if the color and frame rate 
requirements are reduced from real-time true color video, 
it may be possible to implement compressor 16 as a single 
channel, i.e. with a single channel compression subsystem 
29. In this implementation, color data could be compressed 
sequentially for the R, G and B components of the RGB -PGM 
input data, under the control of main controller 28. 



3. 



. Boundary-flpline-wavftlet virion imaaf> riai- a conrore.Qsi nn 



Referring now to Figure 6, a method of compressing 
video image data according to the preferred embodiment of 
the invention will now be described in detail. it is 
contemplated that the architecture of compressor 16 of 
Figure 5 and described hereinabove is particularly suitable 
for the performing of this method, although it is further 
contemplated that other computer architectures and 
arrangements may alternatively be used to perform the 
process of Figure 6. 

The flow chart of Figure 6 corresponds to the 
compression of a single frame of video image data. 
Accordingly, for the compression of a motion picture, the 
process of Figure 6 is performed sequentially for each 
frame of the motion picture. In the case of still image 
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compression, of course, the process of Figure 6 is 
performed for each image. 

The process of Figure 6 begins with the conversion of 
the video image data for the frame into the desired format 
for compression which, according to this preferred 
embodiment, is the well-known PGM format. As discussed 
above relative to Figure 4a, this conversion is preferably 
performed by format converter 14,. allowing compressor 16 to 
be dedicated to the compression process. As noted above, 
the PGM (Portable Grey Level) format expresses each picture 
element ("pixel") as a numerical value corresponding to its 
brightness; for eight -bit PGM, the values range from 0 to 
255. Color images may be expressed in a mode referred to 
in the art as RGB-PGM, where a PGM image is provided for 
each of the red, green and blue color components of the 
image. It is believed that the RGB-PGM format is the most 
adaptable format for processing of color image data by way 
of the present invention, as it directly provides "true- 
color display information for high performance display 
systems, and is also readily adaptable for conversion to 
color look-up table ("color palette") display systems. 

After conversion by format converter 14, the RGB-PGM 
image data is separated into separate R, G, B channels by 
main controller 28. This allows each of channel 
compression subsystems 29 to compress the image data in the 
manner described hereinbelow in parallel with one another. 
Of course, if a monochrome image is being compressed, only 
a single subsystem 29 is necessary; alternatively, if the 
compression rate allows, a color image may be compressed by 
sequentially compressing the separate R, G, B channel image 
data. 
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^ — Boundary- spline - wavelet decompositHrtn 

After channel separation by main controller 28, a 
frame of PGM image data is stored in memory banks 36. in 
subsystem 29 in a row-based order, arranged from top left 
to bottom right of the image. Each PGM image frame is then 
processed by boundary- spline-wavelet decomposition, 
indicated by process 40 of Figure 6. Figure 7 is a 
detailed flow chart of process 40 according to the 
preferred embodiment of the invention, to which attention 
is now directed. The compression method of Figure 7 will 
be described relative to a single channel (R, G, or B) , as 
it will be understood by those of ordinary skill in the art 
that a full color image will be compressed by the parallel 
or sequential compression of the other color channels in 
similar fashion. 

The boundary- spline-wavelet decomposition of process 
40 may operates upon individual blocks of pixels of the PGM 
frame, such as blocks of pixels that may be as small as 
eight -by- eight pixels, or as large as on the order of 1024- 
by-1024 pixels or greater. Process 48 indicates that the 
operation of each subsystem 29 will be performed upon the 
PGM frame in subdivided blocks. One of the important 
benefits of the present invention is that the size of the 
blocks defined in process 48 depends primarily upon the 
architecture of subsystems 29 in compressor 16 and of 
digital matrix processor 30 therein, but is not dictated by 
considerations of picture quality. This is because the 
boundary- spline wavelets used in the decomposition 
eliminate inaccuracies in the displayed data resulting from 
boundary effects. 

Alternatively, if the computing capacity of digital 
matrix processor 30 and subsystems 29 is adequate, an 
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entire frame may be decomposed in process 40 without 
dividing the image into smaller blocks of pixels. The 
present invention will still provide important benefits due 
to the elimination of . boundary effects and inaccuracies 
around the edge of the image that would otherwise be 
present if unbounded wavelets were used. In addition, 
regardless of whether the image is divided into smaller 
blocks, the elimination of boundary effects according to 
the present invention also enables magnification of the 
image to be performed after decompression, without the 
limitations and artifacts that would otherwise be present. 

Following division of the frame into the appropriate 
image blocks, the decomposition of process 40 continues 
with processes 50, 52 in which boundary-spline-wavelet 
decomposition is performed upon the image block, according 
to this embodiment of the invention. As will be described 
in detail herein, according to the preferred .embodiment of 
the invention, the decomposition of process 40 is performed 
utilizing the wavelets described in the Chui and Quak paper 
cited hereinabove and incorporated by reference hereinto, 
and illustrated in Figures 3a through 3d discussed above. 
Process 50 first decomposes each horizontal row of the 
image block into two equal numbers of components of low- 
frequency (low-pass) and high-frequency (band-pass) , and is 
followed by process 52 in which each vertical column of 
each of the low- frequency and high-frequency results of 
process 50 are again decomposed into two equal numbers of 
components of low- frequency and high-frequency. 

Referring now to Figure 8 in combination with Figures 
9a through 9c, decomposition processes 50, 52 will now be 
described in further detail. Figure 8 illustrates a 
frequency domain representation of function f^, and which 
corresponds to image block 51 of Figure 9a that represents 
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the discrete function f N (x,y) . The discrete function 
f N^ x, y' is a functional representation of image block 51 in 
PGM format, such that each value "x" in Figure 9a is a 
digital value corresponding to image brightness at a 
position (x,y). In this example, image block 51 is eight 
pixels square. 

As shown in Figure 8, process 50 of Figure 7 
decomposes^the function into a low-frequency or low-pass 
component f N -i and a high-frequency or band-pass component 
According to the preferred embodiment of the 
invention, process 50 performs such decomposition for each 
horizontal row of image block 51, so that the result of 
process 50 is a matrix 53 that includes, for each row, a 
low frequency portion f H . 1 (x,y) and a high frequency 
portion g N . 1 (x,y). m other words, the decomposition of 
process 50 is performed considering the image data for each 
row of image block 51 as a one-dimensional spatial function 
in the x-dimension. 

As discussed hereinabove, the decomposition of process 
50 is performed according to boundary- spline -wavelet 
decomposition techniques, so that boundary effects are 
eliminated in the resulting frame. Accordingly, for each 
horizontal ^ row of image block 51, the low frequency 
component (or f^^y) in the spatial domain) in the 
corresponding row of decomposed image block 53 is a spline 
interpolation of the spatial data of image block 51. The 
high frequency component g^-l (or g N _ 1 (x,y) in the spatial 
domain) in the corresponding row of decomposed image block 
53 corresponds to the boundary wavelets applied to the 
spatial data of that row of image block 51. 

Referring back to Figure 5, according to the preferred 
embodiment of the invention, it is contemplated that the 
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decomposition of process 50 is performed by digital matrix 
processor 30 by way of a matrix algebra technique. 
According to this matrix technique, the decomposition of 
process 50 is intended to calculate the matrices c N_1 , d N_1 
according to the following relationship: 



[32] 

where c N_1 and d N_1 are matrices of coefficients of the 
low- frequency and high-frequency components of the input 
signal, respectively. 

Attention is directed to Figures 10 . and 11 in 
combination for a detailed description of this technique. 
Figure 10 is a flow chart illustrating the matrix algebra 
procedure for performing processes 50 and 52 of Figure 7 
(the algebra being the same in each case) . Figure 11 is a 
chart illustrating the relationship among the decomposition 
matrices used in performing processes 50, 52 according to 
the flow chart of Figure 10. 

As shown in Figure 10, the first step of the 
decomposition begins with the process 60 which interpolates 
the function f, for each horizontal row, using a B-spline 
function 0(x), to provide a matrix representation {c} N of 
the input image block 51 using the following relationship: 



[33] 
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where m is the degree of the spline, where k is the number 
of sample points within the interval (i.e., the length of 
the row of image^ block 51, in pixels), and where the B- 
spline function <p^ (x) is of the form: 

=2 *$(2J*-k) 

[34] 

According to the preferred embodiment of the invention, it 
has been observed that the accuracy of the decomposition 
and the stability of the result of odd degree splines 
(i.e., linear, cubic, etc.) is improved relative to even 
degree splines (i.e., quadratic). As such, it is preferred 
that m, in equation [31] be an odd number (m = 1, 3, 5, . 
. . ) . 

Upon determining the matrix c N , process 62 (Figure 10) 
is then next performed to compute the coefficients of a 
matrix {a} N , using the relationship a N = C^c 1 *. According 
to this preferred embodiment of the invention, the matrix. 
C 17 is the inner product of the cardinal spline function 

[35] 

The calculation of process 62 returns the matrix a N , as 
shown in Figure 11. 
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Process 64 is then performed in order to compute 
intermediate decomposition coefficients utilizing the 
spline and boundary- wavelet concepts. These intermediate 
decomposition coefficients are also obtained by way of 
matrix algebra using digital matrix processor 30 of 
compressor 16, shown in Figure 5. For the low- frequency, 
low-pass , or spline component , the intermediate 
decomposition coefficients a N_1 are calculated using the 
following matrix operation: 

= (p^-1) T a N 



[36] 



where P 1 *" 1 is a known matrix that contains coefficients of 
the B- spline to be applied to the input image block, and 
which has the following form: 



[37] 



The elements p of the matrix of equation [37] are defined 
as follows: 



k+m-1 dl 



Bl*t k+m-i) 

[38] 

for 1 from -m+1 to -l, and for k from m+21 to -m+l, where 
B ° m ,i represents the B-spline of order m, on the zeroth 
level and with ith initial knot. 
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For the high-frequency, band-pass, or wavelet 
component of the row of the image block 51 , the 
intermediate decomposition coefficients b N " X are calculated 
using the following matrix operation: 



[39] 

i. N-l . 

where Q is a known matrix containing coefficients of the 
boundary-wavelets to be applied to the input image block, 
and having the form: 

[40] 

where the elements q of the matrix are defined as follows: 



J « i 2 



B£ k (k+m-i) 



[41] 



for 1 from -m+1 to -1, and for k from 3m-2+2l to -m+1, and 
where B° m i represents the B-spline of order m on the 
zeroth level and with ith initial knot. In equation [41], 
the values i 2 , i 3 , i 4 are defined as follows: 

1 2 = min (3m-2+21, k+m-2) 

1 3 = max (-m+1, k-m) 

1 4 = rain (2(l+m-l), k+m-1) 
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By way of explanation, the sum of the or terms in equation 
[41] corresponds to the boundary wavelets for those points 
in image block 51 that are near the edges, and the sum of 
the q terms in equation [41] corresponds to the inner 
wavelets for those points in image block 51 that are away 
from the edges. 

According to the preferred embodiment of this 
invention, each of the matrices c", P*" 1 and Q^" 1 are pre- 
calculated and stored in program data memory 35 in each 
channel compression subsystem 29, in a manner accessible by 
digital matrix processor 30, so that the matrix operations 
62, 64 of Figure 10 can be readily and rapidly performed. 
The Appendix to this specification specifies examples of 
these matrices C**, P 1 *" 1 and Q N " 1 which have actually been 
used for decomposition of an image JblocJc ot a size 256 
pixels square, and where the numeric representation of 
matrices C, P and Q in the Appendix, are given for the cubic 
(m=4) case. 

Upon obtaining the intermediate decomposition 
coefficients in process 64, the final decomposition 
coefficient matrices c 11 " 1 and d 1 *" 1 , for the low-pass 
(spline) and band-pass (wavelet) components, respectively, 
are obtained by processes 66, 68, respectively. According 
to this preferred embodiment of the invention which 
utilizes matrix algebra, process 66 obtains the final low- 
pass decomposition coefficient matrix c**" 1 from the 
relationship : 

?? 

where the matrix is the inner product of the cardinal 

spline function 0(x), calculated according to equation [35] 
noted above for the next matrix in sequence. The values of 
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the matrix cN-1 , . . , ^, - 

correspond to the values m the low- 
frequency portion of processed image block 53 of Figure 9b, 

and thus represent the function f„ (x,y) in the spatial 

N-l 

domain. Matrix c therefore also represents the 
5 frequency domain representation f^-1 shown in Figure 8. 

Similarly, process 68 of Figure 10 determines the 
final decomposition coefficient matrix d N-1 for the band- 
pass component of the image block 51. Process 68 is based 
10 upon the operation: 

b 11 " 1 = dN-V" 1 [43] 

where matrix a known matrix that may be 

15 precalculated according to the relationship: 

D"- 1 = (Q N )V +1 (Q N ) [44] 

and is thus based upon known relationships. Upon 
20 completion of process 68, the matrix d**' 1 corresponds to 
the high-frequency values in processed image block 53 of 
Figure 9b, The matrix d N-1 thus also represents the 
spatial function Sjj-i^y)' and * ts corresponding frequency 
domain representation g^, which is the band-pass component 
25 shown in Figure 8. 

Upon completion of the horizontal decomposition of 

N-l N-l 

process 50, matrices c and d are stored in memory 
banks 36 of channel decomposition subsystem 29. preferably 

30 in the order illustrated in Figure 9b as processed image 
block 53. Referring back to Figure 7, process 52 is then 
performed to again decompose processed image block 53 in 
similar manner as process 50, only in a vertical direction, 
i.e., for each vertical column of processed image block 53, 

35 taken column-by- column. In other words, the decomposition 
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of process 52 is performed considering the image data for 
each vertical column of processed image block 53 as a one- 
dimensional spatial function in the y-dimension. According 
to the preferred embodiment of the invention, the matrix 
algebra used in process 52 is identical to that shown in 
Figures 10 and 11, for the next level of matrices in 
sequence, except on a column-by-column basis for each 
vertical column in processed image block 53 . 

Process 52 thus decomposes the low- frequency spatial 
function f N-1 (x,y) into a low- frequency portion "LL" and a 
higher- frequency portion "LH" contained within processed 
image block 55 as shown in Figure 9c, where the second 
decomposition is performed on processed image block 53 in 
a vertical direction. Figure 8 illustrates, in the 
frequency domain, that the matrix portion LL corresponds to 
the function ^-2(^-1) that is the low-pass portion of the - 
prior low-pass function f^-i, and that the matrix portion 
HL corresponds to the function g^-if^-i that is the band- 
pass portion of the prior low-pass function fjj-1. 

Similarly, process 52 decomposes the prior band-pass 
decomposition function g^-l into a low-frequency (i.e., 
lower band-pass) portion ("HL" in Figure 9c J and a high- 
frequency portion ("HH» in Figure 9c) , but where the second 
decomposition is performed on processed image block 53 in 
a vertical direction. Referring to Figure 8, this 
decomposition provides frequency domain representation 
f N' 2 %" 1 corresponding to the HL portion of final processed 
image block 55, and frequency domain representation %-2g N -l 
corresponding to the HH portion of final processed image 
block 55. 

The LL portion of final processed image block 55 
corresponds to the "blur" image of the input video image 



sifci rfuiF Sfftr am p tk\ 



WO 95/19683 



PCT/US95/UU563 



54 

block 51, and the LH, HL and HH portions of final processed 
image block 55 correspond to band-pass portions of the 
input video image block 51, taken in both the horizontal 
and vertical directions, successively. As is evident from 
5 the frequency domain representations of Figure 8, the LH, 
HL and HH components correspond to the components of the 
image in increasing frequency bands, with the HH component 
corresponding to the highest frequency component. 

10 The decomposition performed in processes 50, 52 

described hereinabove provide important benefits in the 
field of video image decomposition. Firstly, the benefits 
of time -frequency localization of the 1 signal in a manner in 
which the frequency-width of the window widens and the 

15 time-width of the window shrinks with increasing frequency 
provided by wavelet decomposition are obtained. This 
allows for more detailed and thorough analysis of rapidly 
changing portions of the signal, and thus a more accurate 
decomposition of the video image. Secondly, by using the 

20 boundary- spline -wavelet approach described hereinabove, 
boundary effects at the edges of the image or of subdivided 
image blocks within the image are eliminated. The 
elimination of these boundary effects allows for smaller 
capacity computers to perform the decomposition with no 

25 degradation in image quality. Elimination of boundary 
effects also enables each subdivided image to be processed 
independently by a stand-alone device or processor system, 
so that the entire decomposition may be performed by 
parallel processing or parallel computing, without 

30 extensive modifications. In addition, the decomposed image 
may, upon decompression, be magnified without the presence 
of boundary effect artifacts in the displayed image. 
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Numerica l decomposition example 

The decomposition of processes 50, 52 can be 
illustrated by way of a simple numerical example. In this 
example, the input image block is eight -by- eight pixels in 
size, and contains an eight -bit digital value 
representative of the display intensity at that location 
(i.e., is in PGM format). An example of the input image 
block in this format is as follows: 
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Process 50, as noted above, performs a boundary- 
spline -wavelet decomposition of the input image block, 
taken for each horizontal row. m this example, where a 
trivial zero degree (m-0) wavelet is used, for purposes of 
explanation, the horizontal decomposition of process 50 
results in the matrix: 
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The portion of the above matrix on the left-hand side of 
the divider corresponds to the low- frequency portion of the 
input image, while the portion of the above matrix on the 
right-hand side of the divider corresponds to the high- 
frequency portion of the input image, both performed in a 
horizontal manner. 



Process 52 then decomposes the result of process 50, 
considering the data in vertical columns as one- dimensional 
spatial sampled functions. Again using the same trivial 
zero-degree wavelet, the vertical decomposition of process 
52 provides the result: 
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The orientation of the above matrix uses the nomenclature 
of Figure 9c, as follows: 
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Video i mage decomposition examp le 

Referring now to Figures 12a through 12e, an example 
of the boundary-spline-wavelet decomposition according. to 
the preferred embodiment of the invention as applied to an 
actual video image will be described. Figure 12a 
illustrates an input video image in PGM format to which 
boundary- spline-wavelet decomposition is applied. Figures 
12b through 12e illustrate the LL, LH, HL and HH components 
of the final processed video image after performing 
processes 50, 52 of Figure 7, using the matrix operations 
described hereinabove relative to Figures 10 and 11. As is 
evident from Figure 12b, the LL component is a quite 
faithful representation of the input image of Figure 12a. 
In the LH, HL, HH components illustrated in Figures 12c 
through I2e, respectively, the visible striations 
correspond to higher frequency components o± the input 
image . 



b. Threshol ding and compression 

Referring back to Figure 6, upon the completion of the 
horizontal and vertical decomposition of the input image 
block performed in process 40, process 42 is next performed 
by quantization processor 32 upon the results of the 
decomposition stored in memory banks 36. 

Specifically, the LH, HL and HH components of, the 
decomposition are subjected to thresholding and 
quantization in process 42, with no such quantization 
performed upon component LL. This selection of the LH, HL 
and HH components for quantization is based upon the 
observation that most real-world images in PGM format will 
consist of low-frequency intensities, and that the higher- 
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frequency wavelet portions of the decomposition will tend 
to have a large number of small or zero coefficients. The 
numerical example set forth above relative to the 
decomposition of the eight -by- eight image illustrates this 
effect. In addition, the exemplary image of Figures 12a 
through 12e also shows that the LL component of the 
decomposition contains the most information, with the LH, 
HL and HH components containing only a small amount of 
significant information indicated by the striations 
therein. 

According to the preferred embodiment of the 
invention, therefore, this large population of small values 
in the higher- frequency components after decomposition may 
be either discarded (or reduced in memory requirements) by 
the quantization of process 42. For example, with 
reference to the above numerical example, all coefficients 
having an absolute value less than 2.0 may be set to zero 
(thresholding) , and all remaining coefficients may be 
rounded to their nearest signed integer value. The memory 
requirements for storage of the coefficients that undergo 
the thresholding and quantization are thus much reduced, 
even before the application of lossless compression 
techniques as will be noted below. 

Other thresholding and quantization techniques may 
alternatively be applied to the decomposed image data, 
selectable according to the code stored in quantization 
data register 31 described hereinabove. 
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Video image quant ization example 

Referring now to Figures 13a through 13c, an 
illustration of the result of thresholding performed- in 
process 42 is illustrated, relative to the video image 
decompression. example of Figures 12a through I2e. Figure 
13a is the same image as the LL -blur" component also shown 
in Figure 12b. Figure 13b illustrates, for example, the 
sum of the LH, HL and HH coefficients after the 
decomposition of process 40 of Figure 7, and thus is the 
sum of the images of Figures 12c through 12e discussed 
above. Figure 13c illustrates the sum image of Figure 13b 
after the application of a threshold limit and quantization 
of process 42. As is evident from Figure 13c, the higher 
frequency components LH, HL, HH from the decomposition of 
process 40 correspond only to the edges and other sharp 
transition locations of the input image. 



£j Completion at video cnmproaai™ 

Referring back to Figure 6, after the thresholding and 
quantization of the LH, HL, HH components of the decomposed 
input image performed in process 42, decision 43 determines 
whether the desired compression ratio has yet been reached. 



As illustrated particularly by the video and numerical 
examples noted above, upon the completion of a single 
decomposition process 40, the LL component of the image may 
be adequate to accurately convey the input image. This 
single decomposition of the image will provide up to a 4:1 
compression ratio, depending upon the memory requirements 
for the quantized high frequency results. This maximum 
ratio considers that the wholesale discarding of the higher 
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frequency components will leave only the LL component 
residing in a matrix that is one-fourth 'the size of the 
input image block. Specifically, therefore, decision 43 
determines if the compression ratio achieved so far is 
5 adequate for the desired transmission or storage and, if 
not, passes control back to the decomposition process so 
that the LL component resulting from the prior 
decomposition may again be decomposed into four more 
components . 

10 

According to this embodiment of the invention, the 
determination of whether the desired compression ratio has 
been obtained may be done relative to a predetermined 
compression ratio. In such a case, timing circuit 37 will 
15 maintain a count of the number of passes through 
decomposition process 40, and will perform decision 43 by 
comparing the resulting compression ratio against a 
previously stored value. 

20 Alternatively, the compression ratio and thus decision 

43 may be determined in a dynamic manner, frame by frame, 
depending upon the accuracy with which the LL component is 
representative of the input frame image. Conceptually, 
such a determination will be a measure of the difference 

25 between the LL component and the input image block relative 
to a predetermined accuracy limit, such that if the LL 
decomposition is within a preselected e limit, an 
additional pass through the decomposition process 40 may be 
performed. It is contemplated that this determination can 

30 be made automatically by main controller 28 in decompressor 
16 so that the process performed in each of channel 
compression subsystems may remain consistent, by 
calculating a numerical value based upon the sum of the 
coefficients in the LH, HL, and HH components of the 

35 decomposed image, such sum indicating the difference 
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between the input image to that pass of the decomposition 
process and the resultant LL component. 

in the usual case, where each video image frame is 
subdivided into smaller image blocks for compression, it is 
preferred that the compression ratio be constant for all 
compressed image blocks of the frame. Accordingly 
decision 43 is preferably performed after each image block 
has been decomposed and quantized in processes 40 ; 42 to 
allow the dynamic determination of whether to repeat 'the 
decomposition process 40 is made based on the worst case 
decomposed image. 

According to this dynamic determination of decision 
43, video image frames that do not contain high frequency 
information to any large extent, such as background images 
with smooth transitions between colors, may be compressed 
to a significantly higher ratio than can video image frames 
that do contain high-frequency information, or a large 
number of sharp transitions. Use of such dynamic 
compression can optimize the overall compression efficiency 
with minimal impact on the image quality. 

Whether statically or dynamically determined, upon 
decision 43 returning the result that the desired 
compression ratio has been obtained, according to the 
preferred embodiment of the invention, lossless compression 
is then performed upon the results ot the decomposed and 
quantized images for the frame, in process 44 of Figure 6 
Referring back to Figure 5, the lossless compression of 
process 42 is preferably performed by lossless compressor 
34 according to a conventional lossless technique such as 
Huffman encoding. The lossless compression of process 42 
is especially beneficial in compressing the quantized 
higher frequency LH, HL, HH components from the 
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decomposition, considering that non-zero or varying values 
in these components will be quite sparse for most video 
image frames. 

After the lossless compression of process 42, the 
compressed video image data is formatted, or coded, on a 
frame -by- frame basis for transmission or storage, as the 
case may be. As noted above, it is contemplated that most 
video image frames will be compressed according to the 
preferred embodiment of the invention after subdivision 
into multiple image blocks, such that a complete trame will 
consist of multiple ones of such blocks, transmitted 
together. As illustrated in Figure 4a, it is preferred 
that the coding of the compressed image data be performed 
within compressor 16, preferably by data flow interface 39 
shown, in Figure 5, prior to its application to digital 
communications network 15. 

Referring now to Figure 14, a schematic illustration 
of formatted frame 70 of data according to the preferred 
embodiment of the invention will now be described. As will 
be described in further detail hereinbelow, the 
decompression of video image data compressed according to 
the present invention can occur very rapidly; for example, 
portions of motion pictures compressed according to the 
present invention have been decompressed and displayed on 
a real-time basis. This rapid and accurate decompression 
allows for the ability of enhanced features to be used in 
the display of the decompressed images. The construction 
of data frame 70 of Figure 14 includes the compressed bit 
stream data corresponding to a video frame, plus the 
appropriate formatting "header" information to allow the 
enhanced features to be operable. The header portion of 
frame 70 preferably provides complete information to 
describe the frame and its position. 
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As illustrated in Figure 14, frame 70 is a sequential 
block of data formatted for a storage device such as 
computer memory, disk storage, CD-ROM and the like it is 
therefore contemplated that, in order to take advantage of 
all of the features of the present invention, compressed 
data frames 70 will be stored in a computer memory prior to 
its decompression, rather than decompressed in a real-time 
manner as received over digital network 15. Real-time 
decompression and display may alternatively be performed as 
the data is received, but certain ones of the features 
described hereinbelow will not be as useful in such a case 
Since the extraction of the header information from frame 
70 requires extremely little computing time and effort 
inclusion of this header information will have 
substantially no penalty in the overall performance of the 
real time decompression and display. 

The first portion of frame 70 is HI header 71, which 
xs data of string type used to specify the status of the 
bit stream compressed image, and as such may contain on the 
order of twenty bytes of data. For example, HI header 71 
may include an identifier of the movie of which the frame 
is a part, and may also contain a security code for 
preventing unauthorized viewing or use of the compressed 
data. For example, if the compressed video data 
corresponds to a movie being communicated over digital 
telephone lines, HI header may contain a code corresponding 
to an identifier of a local decompression system 20 so that 
only that decompression system will be enabled to 
decompress the data. Field 72 of frame 70 is a four^byte 
field of long integer type which contains the number of its 
corresponding frame 70 to specify the position of the frame 
20 in the entire video image sequence. Field 73 is a two- 
byte field of integer type which identifies the group of 
motion (if any) to which frame 70 belongs. a group of 
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frames 70 may be designated as a group, such that the 
frames in the group cannot be separated from one another or 
cut from the entire sequence. Such grouping can prevent 
unexpected side effects in the display of the motion 
picture sequence. 

Fields 74 and 7S then follow in frame 70 according to 
this example, to facilitate control of the display of the 
video sequence containing frame 70. Field 74 is a four- 
byte field of long integer type which contains the address 
at which the previous frame in the sequence begins, 
enabling rapid jumping back to the previous frame as 
desired. As will be described hereinbelow, a user control 
interface may be provided with decompressor system 20 to 
allow interactive control of the display of the video 
sequence, in which case field 74 will facilitate the 
skipping and selection of individual frames in reverse 
order. Similarly, field 75 is a four-byte field of long 
integer type which contains the address of the next frame 
in the sequence, allowing rapid skipping of frames 70 in 
the forward direction during decompression and display. 

Field 76 is a two-byte field of integer type that 
indicates the complexity of the image contained within 
frame 70, by specification of compression ratio, quality 
index, or a user-defined specification of the image, such 
values useful in measuring and controlling the performance 
of the decompression and display. 

Fields 77, 79, 81, 83 indicate certain parameters used 
in the compression of the video data for frame 70. The use 
of header information to communicate these parameters allow 
for decompression system 20 to have selectable 
capabilities, such that it can decompress video image data 
that were compressed according to different techniques, m 
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addition, since each frame 70 includes these fields 77 79 
81, 83, the provision of these fields according to the 
preferred embodiment of the invention enables dynamic 
compression, such that different compression techniques may 
be used on different frames 70 in the same video sequence. 
Field 77 is a two-byte field of integer type that specifies 
the level of decomposition in the compressed data of frame 
70, so that decompression system 20 may perform the proper 
levels of decompression upon receipt. Field 79 is an 
integer field indicating the type of lossless compression 
performed by lossless compressor 34, for example- 



0 
1 
2 



no lossless compression 
Huffman coding 
adaptive Huffman coding 



and so on. Ple ld 81 is a two-byte field of integer type 
that indicates the quantization mode used by quantization 
processor 32 described hereinabove relative to compression 
system 10, for example following the tabular listing of the 
codes noted above. Field 83 is a two-byte field of integer 
type that stores the quantization value used by 
quantization processor 32, and thus depends upon the mode 
indicated in field 81. Using the tabular example noted 
hereinabove for control of quantization processor 32, the 
value of field 83 will indicate the valug ag 

xollows : 
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Mode # (fjp 1d fl1) Field aa ~ r >- 



0 

1 

2 
3 



0 (don't care) 
thresholding value 
dequantizing scalar 
address of the JPEG 
quantization table, 
as an offset from 
the compressed data 
4 address of vector 

quantization table, 
as an offset from 
the compressed data 

As indicated in Figure 14, portion 78 of frame 70 
contains the compressed image data for field 70 and 
follows fields 87, 89. Field 87 is a four-byte fie'ld of 
long integer type that specifies the length of portion 78 
containing the compressed data stream, and is useful in 
assisting the reading of the compressed data stream, such 
as by way of a DMA operation. Field 89 is a four-byte 
field of long integer type that indicates the offset 
distance between field 87 and the start of portion 78 
containing the compressed image data; field 89 may thus be 
used to reserve space within frame 70 for other fields and 
information defined by the user or other developers. 

Portion 78 contains a data stream of the compressed 
video image data for frame 70, as noted above. The length 
of portion 78 will, of course, depend upon the compression 
ratio obtained in the compression process, upon the number 
of pixels in the frame, and also upon the extent to which 
the lossless compression of process 44 compressed the 
higher frequency components of the frame. In addition it 
is contemplated that audio information may also be encoded 
m a conventional manner, and placed within portion 78 of 
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llT"- SUPP ° rt transmis =i- « storage of sound 

motion pictures. 



Ttas arrangement of frame 70 is particularly useful in 

v^eoT aCtlVe deCO " PreSsi ° n — ««Play of a sequence' of 
Video frames. Specifically, fields 74. 75 and 7 6 enable 
decompressor 20 to flexibly display the frames in thT 

fralT^; eSPeCiallX ^ ^ Wh « e "» of 

' ™ r Se,Uential fr — = -"on picture. P or 
decompressor 20 can interrogate field 76 to 

its 7" ^ Pr ° CeSSi ^ «"P-ity of decompressor 20 and 
its display system 2S is such that every frame in the 
seance cannot be decompressed and displayed in real time; 
if so, decompressor 20 can skip to the next frame 70 in the 
seance seated by the contents of field ,s in frame 70 

aSld ^ T " P ° SSible iBa9eS * en ««— 
«Wd. those frames that are not skipped are displayed in 

real-txme, so that the time -dependence of the motion L t£ 
■notion picture is accurately conveyed. 

caoabil"^ ?4 ' 76 alS ° PrOVidS tive display 

capability. as field 74 indicates the address of thl 
previous frame in the sequence. frame ,o allows t^ 
capability of backwards display of a motion picture 

fY"*.**- 70 *" «» ««— or with frames 
pipped as described above based on the time required for 
decompression and the capability of decompressor „. In 
addition, the information provided by field 75 facilitates 
the synchronization of the display of the sequence of 
frames 70, and also allows for easy scaling of the tile 
base to provide slow-motion or enhanced-speed display 

Accordingly. referring back to Figure 6 , che 
formatting or coding of process 46 thus prepares the 
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compressed video image data for transmission over digital 
network 15 for storage, decompression and display by 
decompression system 20, or alternatively to disk unit 22 
for archival storage. 



Construrf ion of f he dsrnm r r af5 p ff r 

^ — Hioh performan ce decompressor 

Referring now to Figure 15a, the construction of 
decompressor 18 according to the preferred embodiment of 
the invention will now be described in detail. it is, of 
course, contemplated that other architectural arrangements 
of circuitry may be used to decompress video data according 
to the present invention. Specifically, it is contemplated 
that a conventional general purpose computer system may be 
capable of performing the compression of the video image 
data according to the present invention. However, the 
example of Figure 15a incorporates a preferred embodiment 
of the circuitry for performing the data decompression 
functions described herein. 

In this example, decompressor 18 preferably includes 
a data flow interface 80 coupled to network 15 to receive 
the incoming bitstream data of the compressed video image, 
as described above relative to Figure 14. Data flow 
interface 80 provides an interface to network 15, and 
serves to separate the R, G, and B components of the 
incoming bitstream into three channels, namely the R- 
stream, G-stream and B-stream channels. Decompressor 18 
also includes main controller 84 and three substantially 
identical channel decompressor subsystems 88 R , 88 , 88 
Main controller 84 preferably controls the operation 'of the 
functional circuitry within decompressor 18, and as such is 
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connected by way of the appropriate bidirectional buses and 
control lines to subsystems 88 and data flow interface 80 
to control the timing, feedback and receiving operation of 
decompressor 18. Main controller 84 is preferably a 
general purpose programmable microprocessor or other 
central processing unit (CPU) of sufficient computational 
power and capacity to process some or all of the image data 
and to control the performing of the image compression 
functions to be described hereinbelow. it is contemplated 
that microprocessors having performance levels similar to 
or greater than those of the 80486 type (including those 
available from Intel Corporation or Cyrix Corporation) , of 
the 68040 type (including those available from Motorola), 
and of the SPARC processor type (available from Texas 
Instruments Incorporated or Sun Microsystems, Inc.) will be 
suitable for use as main controller 84 in decompressor 18. 

The three channel compressor subsystems 88 R , 88 , 88 
each receive a corresponding one of the separated channels 
presented by data flow interface 80, so that the boundary- 
spline-wavelet decompression of the three color components 
is performed in parallel according to this embodiment of 
the invention. The construction of channel decompressor 
subsystems 86 is substantially the reverse of that of 
channel compressor subsystems 29 described hereinabove. 
Each of subsystems 88 R , 8 8(3 , 88 B is therefore constructed 
substantially identical to perform these similar tasks as 
the others, in parallel; for purposes of clarity, the 
following description is provided only for channel 
decompressor subsystem 88 R , it being understood that the 
construction and operation of the other subsystems 88 will 
be similar, if not identical, to subsystem 88_ described 
herein. 
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Each channel decompressor subsystem 88 according to 
this embodiment of the invention is specifically designed 
to perform the functions of lossless decomoression, 
dequantization, and boundary spline wavelet re -compos it ion 
(or decompression) , as used in the decompression operation 
according to this embodiment of the invention. Channel 
decompressor subsystem 88 according to this embodiment of 
the invention includes the circuit functions of digital 
matrix processor 86, timing circuit 107, dequantization 
processor 108, lossless decompressor 82, program data 
memory 112 , and multiple memory banks 110 tor storage of 
image data. 

The incoming data received by subsystem 88, as shown 
in Figure 15a, is received by lossless decompressor 82. 
Lossless compressor 82 may be implemented oy way ot a 
conventional digital signal processor such as the TMSC40 
programmed in such a manner as to perform lossless 
decompression upon the incoming data stream for that 
channel as presented by data flow interface 80. The 
lossless decompression performed by lossless decompressor 
82 may be according to a conventional technique, such as 
Huffman encoding, adaptive Huffman encoding, arithmetic 
encoding, LSQ coding, and the like; specifically, however, 
lossless decompressor 80 must either recognize (from field 
79 of frame 70) or be previously programmed to operate 
according to the lossless compression technique used in 
compression of the incoming data. Lossless decompressor 82 
stores the decompressed incoming image data in memory banks 
110, in preparation for the re -composition of the image 
data. Alternatively, lossless decompressor 82 may be 
implemented as a custom logic circuit for providing this 
function. 
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The output from lossless decompressor 82 corresponds 
to the decomposed image data for that channel (R, g, B) , in 
identical form as that presented to lossless compressor 34 
in compressor IS, as described hereinabove. Dequantization 
processor 108 is a logic circuit or programmable device 
such as a microprocessor, for de-quantize the output from 
lossless decompressor 82. According to this preferred 
embodiment of the invention, dequantization processor 108 
is controllable to operate according to various 
quantization or thresholding modes, according to the 
contents of dequantization code register 109 which receives 
the contents of field 81 of frame 70. The dequantization 
value in field 83 is used in the dequantization performed 
by dequantization processor 108. The results of the 
dequantization are stored in memory banks 110. 

Digital matrix processor 86 performs the boundary- 
spline -wavelet composition upon the results of the 
dequantization, . using pre -calculated matrices stored in 
program data 112 . The procedure for this composition is 
described in detail hereinbelow. The results of the 
operations performed by digital matrix processor 86 are 
stored in memory banks 110. As in the case of compressor 
16, digital matrix processor 86 may be implemented by way 
of a conventional digital signal processor, such as the 
i860 microprocessor available from Intel Corporation and 
the TMSC40 digital signal processor available from Texas 
Instruments Incorporated, or by way of a general purpose 
microprocessor such as those of the 80386 and 80486 
families available from Intel Corporation, and of the 68030 
and 68040 families available from Motorola. 

Timing circuit 107 performs the functions of 
controlling the storage of the channel data from lossless 
decompressor 82, and the subsequent retrieval thereof for 
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dequantization and decompression processes. Upon 
completion of the decompression of the channel data for a 
frame, timing circuit 107 is operable to present the 
decompressed PGM channel data to main controller 84, for 
formatting and transmission to format converter 24 for 
display at display 26. 

Main controller 84 receives the decompressed channel 
data from each of subsystems 88 R , 88 G , 88 B and sequences 
the data into the format suitable for receipt by format 
converter 24. In addition, main controller 84 operates to 
provide feedback to data flow interface 80, such that upon 
decompression of a frame of data, data flow interface 80 
can be controlled to communicate the next frame of data to 
subsystems 88 for decompression. 

In addition, main controller 84 is coupled to user 
interface device 85, which may be a mouse, trackball, pen 
input,- keyboard or the like; in addition, user interface 
device 85 may be implemented as another computer, such as 
a personal computer workstation, through which a user 
controls the decompression and display of video data. As 
noted hereinabove, the construction of frame 70 shown in 
Figure 14 facilitates the interactive control of the 
decompression process, such that the user may control the 
direction (forward/backward) and rate of decompression and 
display of the sequence of video image frames, using user 
interface device 85. Main controller 84 is thus operable 
to communicate to data flow interface 80 the order, and 
selection of frames for decompression; it is therefore 
useful to incorporate adequate memory for the storage of 
multiple frames either within or accessible by data flow 
interface 80 to allow such interactive decompression and 
display. 
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the ell ^pressor l 8 ls intended to support 

the concession of high definition real- time true ZlZ 
video lma3e data, where -true color" indicates the use of 

results ln l 6 . 7 mlnion ^ 
dec 0mp ion race for decompressor ^ ^ ^ 

implementation, color data could * " 

sequentially for the R G L T decompressed 
con , rnl . . ' G and B components under the 

control of main controller 84 t« m **>*. 

perform the lossless decompression, as well. 

The construction of decompressor i 8 d^^K „ 

"^1 " hi ~ ~ -™ 

a the Mlth ^ hi ^-anc. workstation eu h 
as the 4D series workstations available fr™, <=■,. 
graphics, and SPARC workstations ^J^Z lZ 

aae"TT' M th6Se »— PerforJceTevet 

adequate for delivering „ore than thirty fran.es per selond 

ZJt :°:T ti0 * <80 ° PlXelS * 600 ^> ^ 9 e ^ a 
with 24-bit true color capability. 
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^ ^sona! -compter ^ ard „ n 

system 

Referring now to Figure 19, decompression system 200 
according to an alternative embodiment of the invention 
will now be described in detail. Decompression system 200 
is mtended for implementation on a conventional PC-based 
workstation, and is configured as a single 8-bit expansion 
board; alternatively, system 200 may be configured as a 16- 
bit ISA or EISA expansion board, or as a MCA (Micro Channel 
Architecture) board, m each case, it is contemplated that 
decompression system 200 will be suitable for medium 
resolution displays. 

Decompression system 200 includes lossless 
decompressor 202, which is implemented as a video 
processing board. Lossless decompressor 202 has an input 
port coupled to network 15 (or, of course, to disk storage 
or another source of video image data compressed according 
to the method described hereinabove). Lossless 
decompressor 202 includes digital signal processor (DSP) 
204 whxch is a conventional digital signal processor such 
as the TMSC25 and TMSC30 types available from Texas 
instruments Incorporated, or a general purpose 

TrZTT e T r 8UCh M 68020 ° r 68030 ^ e liable 

from Motorola. Program memory 206 is preferably EPROM or 

other conventional firmware storage, for storing the 
programs utilized by DSP 204 in performing the lossless 
decompression upon the received data. 

Lossless decompressor 202 receives the compressed RGB- 
PGM data from, network is and performs the necessary 
lossless decompression according to the lossless 
compression technique used (if any) in compressor 16 it 
is preferred that lossless decompressor 202 be capable of 
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performing decompression of various types, selectable by 
way of field 79 in frame 70, described hereinabove relative 
to Figure 14. Lossless decompressor 202 is also coupled to 
user interface device 205, to allow a human user the 
capability of controlling motion control in the 
decompression and display of a sequence of video images. 
Lossless decompressor 202 presents its output to 
decompression processor 210 for the boundary- spline -wavelet 
decompression . 



Decompression processor 210 according to this 
embodiment of the invention includes processing capability 
to perform boundary-spline-wavelet decomposition according 
to the preferred embodiment of the invention described 
herein. The main controller of decompression processor 210 
is digital signal processor (DSP) 225, which is preferably 
a high-performance single-chip digital signal processor, 
such as the TMSC40 available from Texas Instruments 
Incorporated or the i860 processor available from Intel 
Corporation, or a high-performance general purpose 
microprocessor such as the 80486 microprocessor available 
from Intel Corporation. Decompression processor 210 also 
includes program data memory 212 for storage of program 
code and pre -calculated matrices useful in the 
decompression process, data memory banks 214 for storage of 
image data, and digital matrix processor 220 for performing 
matrix operations useful in the decompression routine. 
Digital matrix processor 220 is preferably a conventional 
digital signal processor, such as the i860 microprocessor 
available from Intel Corporation and the TMSC40 digital 
signal processor available from Texas Instruments 
Incorporated, or a general purpose microprocessor such as 
those of the 80386 and 80486 families available from Intel 
Corporation, and of the 68030 and 68040 families available 
from Motorola. Host interface controller 208 is also 
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provided within decompression processor 210 for 
communicating with a host computer (not shown) including 
the video display associated therewith. Program memory 222 
is also provided in decompression processor 210 and 
contains the program code for boundary spline wavelet 
reconstruction and format conversion. Video timing control 
circuit 217 is a i so contained within decompression 
processor 210, for controlling the viewing of the 
decompressed data as well as the timing of the 
decompression, so that a sequence of video frames may be 
vxewed in real-time as a movie, or under the control of 
user interface device 205. 

In operation, lossless decompressor 202 stores the 
results of the lossless decompression for a frame in memory 
banks 214. These data are retrieved by DSP 225 via memory 
interface 224, according to a sequence stored in program 
memory 222, and forwarded to digital matrix processor 220 
along with precalculated matrices stored in program data 
212 along the coefficient bus shown in Figure 19. Digital 
matrix processor 220 performs matrix operations upon the 
image data presented by DSP 225 to perform the boundary- 
splme-wavelet decomposition and forwards the results to 
DSP 225 along the data-out bus shown in Figure 19 for 
storage in memory banks 214. Digital matrix processor 220 
also provides control information to DSP 225 on the 
feedback bus therebetween. 



Upon completion of the boundary- spline -wavelet 
reconstruction or decompression, the decompressed image 
data for a frame is stored in memory banks 214. Memory 
interface 224 enables the interleaved or simultaneous 
accessing of memory banks 214 for boundary-spline-wavelet 
decompression and display; host interface controller 208 
accesses memory banks 214 for display purposes. Host 
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interface controller 208 may also have sufficient 
capability to directly display the contents of memory banks 
214 containing the decompressed video image data- 
alternatively, the decompressed image data may be forwarded 
by host interface controller 208 to memory of the host PC 
for eventual display, by way of DMA. 

It is contemplated, of course, that still further 
alternatives to the decompression hardware may be utilized 
according to the present invention. 

S, Pounds-spline.^ pf „ ld » n imafTm 

and dl«m» Y 

Referring now to Figure 16, a method according to the 
preferred embodiment of the invention, for decompressing 
video xmage data that was compressed by way of boundary- 
spline-wavelet compression as discussed above, will now be 
described m detail. This decompression will be described 
relative to the hardware decompression. system of Figures 4a 

wf J 53 ,; ^ bSing UnderSt °° d that analogous operations 
would be carried out by decompression system 200 
illustrated in Figure i 9 , for its particular 
implementation . 



The initial step of the decompression method of this 
example is the reading of the compressed file or sequence 
of frames 70 by data flow interface 80, indicated in Figure 
16 by process 90. Process 90 may be the result of the 
receipt of communicated compressed video image data by data 
flow interface 80 directly from digital network 15 
Alternatively, where decompressor is is a portion of a 
stand-alone computer, disk unit 22 or another fixed storage 
unit may contain the file of compressed video image frames 
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from an earlier transmission, or as archival storage m 
either case, the result of process 90 is the receipt of a 
sequence of compressed video image frames ,o by data flow 

Figure 16 can retrieve individual frames 70. 

Process 91 is next performed. where each of the 
channels of a selected frame 70 n undergoes lossless 

Compressed in^a^ e^rd^T /he Preferred 
embedment of the invention, and as such the process fT™ 

a si:r e h s f T this *** fotwara " U1 be ^ 

» "ngle channel <E, G, B) of frame ,<,. The particular 

"Z ne iS t r l8Cted aCC ° rdln3 E ° «*— <* -me. 
the user v PiCtU "' " al "™"vely selected by 

the user via user interface device 85. In either case 

To so™" 84 em ' the ~— ° f 

™„. so that the appropriate frame 70 is presented to 

Channel decompressor subsystems ... which provTde the d at a 

for frame 7 0 n to lossless decompressor ... noted ahove 

the lossless decompression performed in process 91 bv 

loss es decompressor 82 corresponds Vthe type of 

lossless compression performed by lossless compressor^* ll 

compressor «. ,he result of the lossless degression of 
proces, ^ provides ^ finai deco ^ sed £ 3ion of 

relative »' " 

relative to the compression method. 

After the lossless decompression of process 91, the 
image data is deouantized for each channel R, o, B % 
deouantization processor l„s in its corresponding channel 

P=rforme e d 8 b "d SUbSySten ° f ^-"za^n 

performed by denization processor 10. is controlled by 

the contents of denization data register 109, whij 
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contains a value corresponding to that transmitted as field 
81 in frame 70 n . In the event that the data was quantized, 
the quantization value transmitted in field 83 of frame 70 
is utilized to restore . the data to that prior to 
quantization in compressor 16. 

Upon completion of the dequantization of process 92, 
process 93 is performed for each image block of frame 70 ' 
to reconstruct the full video image in PGM format, with 
each pixel location containing a value corresponding to the 
intensity at that location. In effect, process 93 reverses 
the decomposition process described hereinabove relative to 
Figures 6 and 7, according to the boundary- spline -wavelet 
approach. This reconstruction of process 93 is performed 
on a block-by-block basis, for each of the image blocks in 
the selected frame 70 n . Upon completion of process 93, the 
selected frame 70 n is stored as a PGM format frame in 
memory banks 110. 



Referring now to Figure 17, a method for 
reconstructing the selected frame 70 Q according to the 
preferred embodiment of the invention will now be described 
in further detail. Recalling that the decomposition of the 
video image data performed during compression is performed 
first in the horizontal direction and then in the vertical 
direction, the decomposition must be performed in the 
reverse order. The first step in the reconstruction of the 
video image data is therefore the reconstruction of each 
column in the vertical direction, as indicated by process 
98. 

The reconstruction process -98 is also most 
efficiently, and thus preferably, performed by digital 
matrix processor 86 in the corresponding channel 
decompression subsystem 88 using matrix algebra operations 
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similarly as i„ the egression process described 
hereinabove. Accordingly, process 9 a begins by consider^ 
the image block in columns, each containing a set of 

* b ina "he""/ "* frS ™ P °" ion ' 

If 3 nUB,ber ° f tiMS th « »°undary-spline-wavelet 
decomposition was performed during concession to obtaL 

olressi =0mPreSSi ° n ra "°- FM ™- «— ^ 
expression ratl o was 4=!. the value of k would eoual a 

once)': r!r / ' deC °"*° si "°* — Perform 

once), for a compression ratio of 64=1, k would equal e: 

used ST " U1UStrateS tne -l^ionship of the matrices 
Zees" 3 "instruction of the image block performed in 
directed to ^^^^ ^ration (process ,., is 

values ! 3 ^ ^ ^ ° rder ™«icient 

N-k+1 

ma ^. ; Conversel y to the case of compression, the 

matnx operation is equivalent to: 



c N-k + l = pN -k c N-k + Q N-k d N-k 



[45] 



»h«e the matrices P* and oF are the spline and wavelet 
-trices respectively, for level N, as used in the 
decomposition and compression process. As not ed Lve 
matrices P « and gP can be pre-calculated and stored itThe 
Program data memory « of channel decompression subsystl 

Process 9a thus performs one level of the matri* 
operati o£ equation M} ^ , co 

each column in the image block, tfcon completion of P ™ 
«. Process „. performs a similar reconstruction as tlat 
in process ... only on a row-by-ro„ basis for each row^n 
the image block. As such, the operation of equation^ 
Performed to obtain the coefficients c^ 1 " 2 . 
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It will, of course, be noted that the decompression of 
the video image data according to the preferred embodiment 
of the invention will not result in an exact duplicate of 
the input image presented . to the compression process, as 
the effects of the quantization and thresholding process in 
the compression of the video image data cannot be full 
recovered (i.e., some data is lost). As such, the 
boundary-spline-wavelet compression according to the 
present invention is a lossy compression technique, it is 
contemplated, however, that the use of the boundary- spline - 
wavelet approach described hereinabove causes the loss of 
only that data at high-frequencies, such that the loss of 
quality of the image is minimized for a given compression 
ratio. 



However, it should especially be noted that the 
decompression processes 98, 100 can be done quite rapidly 
particularly with specialized circuitry such as a digital 
matrix processor performing the matrix operations noted 
above. This speed in processing arises from the wavelet 
forms selected in the compression, which correspond to 
express mathematical formulae and which are also quite 
adaptable to matrix operation techniques. As such, it is 
contemplated that the decompression performed by 
decompressor 18 can occur quite rapidly and, for many 
applications, need not require high performance 
microprocessor circuitry. 

Upon the reconstruction in both the column and row 
directions as performed in processes 98, 100, decision 101 
is performed to determine if the image block has been fully 
reconstructed (i.e., if the image block corresponds to a 
PGM format image, and does not contain low and high 
frequency components) . if not, the matrix operations of 
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processes 98, 100 are repeated to perform the next levels 
of reconstruction of the image for that image block. 

Upon process 100 completing the reconstruction of the 
image block, decision 103 tests whether more image blocks 
remain to be reconstructed in the selected frame 70 if 
so, the next image block is retrieved from memory 80 
(process 104) and is reconstructed by processes 98, loo 
101 as before. Upon completion of the reconstruction of 
all image blocks in selected frame 70 n , process 106 is 
performed so that the reconstructed image blocks in frame 
70 are stored in image buffer 88, arranged in the proper 
order to correspond to a full PGM format image frame 
completing process 93 of Figure 16 for a single frame 70 .' 
The decompressed image data for the frame 70 n , including 
that for all three channels R, G, B, are the forwarded t! 
main controller 84 for communication to format converter 
24. Main controller 84 then indicates completion of a 
frame to data flow interface- and performs decision 95 to 
determine if additional frames are to be decompressed and 
display; if so , main controller 84 will select the next 
frame for decomposition in the sequence, or as directed by 
user interface device 85. 

Referring back to Figure 16, process 94 is then 
performed by way of which format converter 24 reformats and 
communicates the frame to display 26, in the desired 
fashion. It is contemplated that format converter 24 may 
perform such operations as gathering three successive, PGM 
format frames, corresponding to the red, green, and blue 
color components of the same image, and presenting the data 
in RGB -PGM format to display 26; format converter 24 may 
also convert the RGB -PGM format data into another format 
such as PCX, IMG, GIF, TIF, RLE, ntsc and the ^ ^ 
appropriate for display 26. 
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Referring now to Figure 15b/ the construction of 
format converter 24 according to the preferred embodiment 
of the invention will now be described in detail. Format 
converter 24 includes three buffers 1 20r , iao c , 120 D for 
receding from decompressor IS and storing PGM format data 
corresponding to the three decompressed R, g, B channels 
The outputs of buffers 120 are received by color combiner 
Circuit 122, which is a data processing circuit such as a 
microprocessor, which combines the decompressed video data 
for tne three R, G, B channels into RGB-PGM format, color 
combiner 122 presents the results of this combination to 
the appropriate formatting circuitry for driving display 
26 m the analog case, color combiner 122 presents its 
• output to video digital-to-analog converter (DAC) 126- 
vxdeo DAC 126 is a conventional video DAC suitable for 
receiving digital video or graphics information and driving 
ana! g di lay 26a . ^ ^ ^ ^ ^ ^ng 

122 preeents lts output to color reducer ^ forraafc ^ 

124 Reducer/encoder 124 is a conventional circuit for 
encoding the RGB -PGM format data presented thereto by color 
combiner 12 2 into the desired format for digital display 
26d, such formats including TIF, IMG, GIF and the like Of 

rTpIm if /T ay ^ 13 CaFablS ° f di —^ ^splaying 
RGB -PGM video data, reducer/encoder 124 is unnecessary 

ar S J 11 additi ° n ' f0rmat inverter 24 may also include 
graphics processing capability so as to perform more 
graphics processing operations, including magnification 
rooming and the li ke . A s noted above, the elimination of 
boundary effects resulting from the boundary- spline -wavelet 
compression and decompression according to the present 
invention is especially beneficial when such complex 
graphics processing features. For example, a portion of 
the image of frame 70 n may be readily magnified according 
to conventional graphics processing techniques, such as the 
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use of interpolation techniques to fill in pixels between 
those for which the frame contains actual data, with a high 
degree of accuracy. m contrast, boundary effects that 
result from prior compression techniques tend to be 
exaggerated when the image is magnified, considering not 
only the magnified display but also the exaggeration 
resulting from interpolation between "true" image data and 
the boundary effect artifacts. 

Indeed, it is contemplated that the present invention 
wxll be especially beneficial for the transmission and 
storage of compressed video images for display on advanced 
dxsplays that have higher pixel densities than the density 
of the image so compressed, due to the elimination of 
boundary effect artifacts by the present invention. m 
add-on, it is contemplated that such magnification will 
allow for the display of the images on a plurality of video 
displays arranged in an array, with each display showing 
only a portion of the full image. 

Referring back to Figure 16, upon transmission of the 
selected frame 7<> n to format converter 24 performed in 
process 94, decision 95 is then performed to determine if 
additional frames in the sequence are to be decompressed 
for display, if SO/ process 96 selects the next frame 70 
for decompression and display. According to the preferred 
embodiment of the invention, the selection of process 96 
may be made in multiple ways. a first approach for 
selecting the next frame for decompression and display is 
merely to select the next frame in the sequence, and access 
the same by way of field 75 (Figure 14) from the previously 
decompressed frame 70^ this approach will be followed 
during the decompression and display of a motion picture 
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As noted above relative to Figure 14, field 76 

ratiTL l indiCa " nS SUCh £a=t0rS M "» ^ssion 

comoLT tha * indiC " e the ^"ational 

complexity required for the decompressor, and display of 

next 7" all °" S £ ° r intelli *=" "iection of the 

next frame in process 96, wher e the capability of 
decompression system 20 is limited relative to the amount 
of compressed video data in the transmission. For example, 
if the time that decompressor 18 win require for 
decompression and display and display of a frame is longer 
than the reciprocal of the frequency at which frames are to 
be displayed (e.g., longer than !/30 sec. in the case of a 
conventional motion picture) , main controller 84 will skip 

decor' " e " framSS ' Md SeU « * *« 

^ » «-t the sequence of frames 

that are displayed will appear in real-rim. * >.• 
display 26. real-time fashion on 



in J „ f ° r tha Bele « i! » °« the next frame 

in process 96 is that which may be directed by the user via 
user interface device 65. For example, the viewer may wish 
or i« y ' 3 basis JZt 

backed direction, or the viewer may wish to repetitively 
view a eelected portion of the sequence. These commands 
are conveyed to main controller 84 by user interface device 
85, and are used in process 96 according to the present 
invention to select the next frame. . 

Opon the decompression and display of all of the 
frames in the sequence as determined by decision 95, the 
process of decompression and display ends 
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5 . Conclusinn 



The methods and systems for compressing and 
decompressing video image data described hereinabove 
relatxve to the present invention provide important 
advantages, as noted throughout the • foregoing 
speculation. These advantages include the fundamental 
benefits of wavelet analysis, where time- frequency 
localization of the input signal is implemented so that the 
t lme window narrows with increasing frequency and widens 
with decreasing frequency, thus providing highly accurate 
analysis for transient periods of the signal, and avoiding 
the undersampling problems of conventional Fourier analysis 
techniques. 

In addition, the video i mage compression and 

eLoTT° n teChniqU6S aCCOrdin3 t0 the Purred 
embodiments of the invention utilize boundary- spline- 
wavelets to eliminate boundary effects in the compressed 
images, specifically by using different wavelets for 
samples near the boundaries of an interval from those 
wavelets used for inner samples. The boundary wavelets do 
not require support from outside of the interval; when 
applied to video compression according to the present 
invention, therefore, boundary effect artifacts are 
eliminated. This not only provides an extremely accurate 
representation of the original input image but, when 
coupled with the computational geometrical superiority of 
spline functions, also enables the performance of complex 
processing operations and display such as magnification 
dynamic compression on a frame-by-frame basis, interactive 
dxsplay of a motion image, including insertion, editing and 
repetitive display. j» addition, the present invention 
also enables slow display systems to skip frames so that a 
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motion picture can be displayed on a ~i - ■ u 

fewer F-^yea on a real-time basis with 

tewer trames per second. 

The elimination of boundary effects also enables th e 
5 compression routine to be i ■ ena *>J-es the 

parallel processing techniques to be readily applied to til 

while still oh,, T impression and decompression. 
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in addition, the present invention also enable „ 
Ration pr ovlded by ~ ZLZZLZZSZZZ 

enaoj.es the dynamic compressinn 

™-=^ 

The present invention may be amU^i - 

applications of video ima ^ ° Vari ° US 

viaeo xmage compression, such » = 

transmission of motion pictures and other vide! 
between remote locations, such a ^ 
c = cation network or by satellite transTssion 
present invention may also be app lie d advantageously to T 
archival storage of video image data, both TtTonXur 
and stm ln ,aae data, in a compressed for*. The archivll 

acilitated by the present invention due to the lack J 
Boundary effects, as the decompression and display of tne 
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compressed information may be performed in much higher 
density and capability systems than those used in the 
original compression, without exaggeration of artifacts 
such as boundary effects. 

While the invention has been described herein relative 
to its preferred embodiments, it is of course contemplated 
that modifications of, and alternatives to, these 
embodiments, such modifications and alternatives obtaining 
the advantages and benefits of. this invention, will be 
apparent to those of ordinary skill in the art having 
reference to this specification and its drawings. it is 
contemplated that such modifications and alternatives are 
within the scope of this invention as subsequently claimed 
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WE CLAIM: 

1. A method of compressing digital data 

representative of a video image for communication thereof, 
5 comprising the steps of: 

formatting the digital data corresponding to a 
frame of the video image into an array of locations in rows 
and columns, with the value at each array location 
corresponding to the intensity of the display; 
10 decomposing, in a first direction, each row of 

said array into low-frequency and high-frequency portions 
using boundary-spline-wavelets, and arranging the results 
of the decomposing into corresponding rows; 

then decomposing each column of said arranged 
15 results of the first decomposing step in a second 
direction, using boundary-spline-wavelets; and 

communicating the results of the decomposing 
steps to a receiving unit. 

20 2. The method of claim 1, wherein said communicating 

step comprises: 

storing the results of the decomposing steps in 
a fixed memory. 

25 3. The method of claim l, wherein said communicating 

step comprises: 

transmitting the results of the decomposing steps 
to a decompression system for display. 

30 4. The method of claim 3, further comprising: 

after receiving the transmitted results of the 
decomposing steps, reconstructing the transmitted video 
image in the second direction, using boundary-spline- 
wavelets; 
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then reconstructing the results „ f -k ^. 
reconstructing step i„ the f irst di * 
spline-wavelets; and ^ bo ^ry- 

- steps on :illl7Z£: ° f ^ —cting 

10 freguency <»J^ Z^'*™"* 0 ^ t0 l0W " 

first decomposing step; direction of the 

frequency eo J^ ' =™o M1 „ g to the iow- 

"igh-freguency deco^ion in^T* dlreCtl0 "' ° f 
» ««t decomposing .^LT ' """"" °* «» 

freguency comp^eT t^l 0 T: h COrreSPOndin9 <° 

first decomposing step; Erection of the 

25 and further comprising; 

.. The *w , of elila „ f ^ 

». HL, and HH components —•«—»« on the 

7. The method of claim 5, further comprising; 
repeating said decomposing steps. 
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8. The method of claim 5, further comprising: 

transmitting the LL component and the quantized 
LH, HL, HH components over a communications network; 

receiving the transmitted components from the 
5 communications network; 

reconstructing a low-frequency component from the 
LL and quantized LH components using boundary-spline- 
wavelets in the second direction, and reconstructing a 
high-frequency component from the quantized HL and HH 
10 components using boundary-spline-wavelets in the second 
direction; 

then reconstructing a video image from said low- 
frequency component and said high-frequency component using 

boundary-spline-wavelets applied in the first direction; 
15 and 

displaying the results of the reconstructing 
steps on a video display. 

9. The method of claim 8, further comprising: 
20 after said quantizing step and prior to said 

communicating step, performing lossless compression on the 
LL, LH, HL, and HH components; and 

after said receiving step, performing lossless 
decompression on the communicated LL, LH, H L and HH 
25 components. 

10. The method of claim 8, further comprising: 
repeating said decomposing steps a selected 

number of times prior to said communicating step; and 
30 repeating said reconstructing steps the selected 

number of times prior to said displaying step. 

11. The method of claim 5, further comprising: 
prior to said decomposing steps, dividing each 

35 frame into a plurality of image blocks; 
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wherein said decomposing steps are performed for each 
image block of each frame, so that each of said image 
blocks comprises a ll component and quantized LH, hl, and 
HH components. 

12. The method of claim ll, further comprising: 

transmitting the LL component and the quantized 
LH f HL, HH components for. each of said image blocks of each 
frame over a communications network; 

receiving the transmitted components from the 
communications network; 

for each of said image blocks, reconstructing a 
low-frequency component from the LL and quantized LH 
components using boundary-spline-wavelets in the second 
direction, and reconstructing a high-frequency component 
from the quantized HL and HH components using boundary- 
splme-wavelets in the second direction; 

then, for each of said image blocks, 
reconstructing a video image from said low-frequency 
20 component and said high-frequency component using boundary- 
spline-wavelets applied in the first direction; 

arranging the reconstructed video image for each 
of said image blocks into a video frame; and 

displaying the video frame on a video display. 

25 

13. A method of displaying compressed video image 
data, comprising: 

receiving compressed video image data 
corresponding to a frame and storing the compressed frame 
30 ln memory, said compressed video image data arranged as 
first, second, third and fourth frequency domain 
components ; 



in a 



first image direction, applying boundary- 
spline-wavelets to each column of data in the compressed 
35 frame to reconstruct a low-frequency component from the 
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first and second frequency domain components, and to 
reconstruct a high-frequency component from the third and 
fourth frequency domain components; in a second 

image direction, applying boundary-spline-wavelets to each 
row of data of the low-frequency component and high- ' 
frequency component to reconstruct the video image frame- 

displaying the reconstructed video image frame on 

a vadeo display. 

10 14. The method of claim 13, further comprising: 

repeating said steps of applying boundary-spline- 



- — i-jr— j — »<g */wuuwiaiy— spin 

Z^T! " first Md second toa " 

selected number of times. 



a 



■rn.iv- . aeth ° d ° f Claim 13 ' Wherein said ^eps of 
applying boundary-spline-wavelets are performed, for each 
frame, for a plurality of image blocks of said frame. 

16. The method of claim 13, further comprising: 
performing lossless decompression on the 

compressed video image data, prior to said applying steps! 

17. The method of claim 13, further comprising- 

25 r~ «. aft6r applyin * ste P s * storing the 

25 reconstructed video image frame in an image buffer. 

18. The method of claim 17, further comprising: 
after said storing step, magnifying the image of 



30 



said video image frame prior to said displaying step." 

\«* ^ I- A SySten f ° r comnunic ^ing video image 
information, comprising: 

an input source for providing digital video image 
information arranged as frames; and 
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compressor circuitry having an input coupled to 
said input source, for decomposing each frame of digital 
video image information in a first image direction using a 
boundary-spline-wavelet into a first low-frequency 
5 component and a first high-frequency component, and for 
then further decomposing each of said low-frequency and 
high-frequency components of each frame of digital video 
image information in a second image direction using a 
boundary-spline-wavelet into first and second pairs of low- 
10 frequency and second high-frequency components, said first 
pair being the decomposed representation of said first low- 
frequency component and said second pair corresponding to 
a decomposed representation of said first high-frequency 
component, said compressor circuitry also having an output 
15 for presenting the first and second pairs of low-frequency 
and high-frequency components. 

20. The system of claim 19, further comprising: 

a format converter circuit coupled between said 
20 input video source and said compressor, for converting the 
digital input video information into a portable gray-level 
format . 

21. The system of claim 20, wherein said input source 
25 provides color video information; 

and wherein said format converter circuit is for 
converting the color video information into a plurality of 
portable gray-level format representations, each 
representation corresponding to a color component. 

30 

22. The system of claim 19, wherein said compressor 
circuitry comprises: 

decomposing circuitry for performing the 
decomposing of each frame of digital video image 
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information into the fircf = _ j « , 

tile iirst and second pairs of low- 
frequency and high-frequency components; 

quantization circuitry for quantizing the high- 
frequency component of said first pair, and for quantizing 
5 the both the low-frequency and the high-frequency component 
of the second pair; and 

lossless compression circuitry, having an input 
coupled to said quantization circuitry, for performing 

10 IZ l S \ C T T&SSim ° f ^ ^ S6COnd ^"-frequency 

10 and high-frequency pairs, prior to presenting the 

decomposed frames of digital video image information at the 
output of the compressor circuitry. 

_ 23. The system of claim 22, further comprising: 

mam controller circuitry, for controlling the 
operation of the decomposing circuitry, so that the low- 
frequency component of said first pair may be repetitively 
provided to the decomposing circuitry for further 
decomposing. 

20 

24. The system of claim 19, wherein said compressor 
circuitry comprises: 

an image buffer coupled to the input of the 

25 ClrCUitry f ° r St ° rin * a ***** ot digital video 

25 image information; and 

a digital matrix processor for performing the 
decomposing of each frame of digital video image 
information into the first and second pairs of iow- 
frequency and . high-frequency components using a boundary- 
30 spline-wavelet. 

25. The system of claim 24, wherein said digital 
matrix processor operates according to matrix operations 
using precalculated matrices corresponding to spline and 
35 wavelet components and stored in said compressor circuitry 
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10 



26. The system of claim 19, further comprising: 

a storage unit, coupled to the output of said 
compressor circuitry, for storing the decomposed frames of 
digital video image information. 

27. The system of claim 19, wherein the output of 
said compressor circuitry is coupled to a digital network 
for communication of said decomposed frames of digital 
video image information. 



28. The system of claim 27, wherein said compressor 
circuitry comprises: 

decomposing circuitry for performing the 
decomposing of each frame of digital video image 
15 information into the first and second pairs of low- 
frequency and high-frequency components; 

quantization circuitry for quantizing the high- 
frequency component of said first pair, and for quantizing 
the both the low-frequency and the high-frequency component 
20 of the second pair; and 

lossless compression circuitry, having an input 
coupled to said quantization circuitry, for performing 
lossless compression of the first and second -low- frequency 
and high-frequency pairs, prior to presenting the 
25 decomposed frames of digital video image information at the 
output of the compressor circuitry; 

and wherein the output of the lossless compression 
circuitry is coupled to the digital network so that the 
30 decomposed frames of digital video image information are 
communicated after the lossless compression. 

29. The system of claim 19, further comprising: 

a communications network coupled to the output of 
35 the compressor circuitry; 
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decompressor circuitry, having an input coupled 
to said communications network, for reconstructing each 
decomposed frame of digital video image information in the 
second image direction using a boundary-spline-wavelet, and 
5 for then further reconstructing each frame of decomposed 
digital video image information in the first image 
direction using a boundary-spline-wavelet; and 

a video display, for displaying the reconstructed 
frames of digital video image information. 

10 

30. The system of claim 29, wherein said compressor 
circuitry comprises: 

decomposing circuitry for performing the 
decomposing of each frame of digital video image 
15 information into the first and second pairs of low- 
frequency and high-frequency components; 

quantization circuitry for quantizing the high- 
frequency component of said first pair, and for quantizing 
the both the low-frequency and the high-frequency component 
20 of the second pair; and 

and wherein said decompressor circuitry comprises: 

reconstructing circuitry, for reconstructing a 
digital output video frame from the decomposed frames of 
digital video image information, by reconstructing an 
25 approximation of the first low-frequency component from the 
first low-frequency and high-frequency pair in the second 
image direction, and an approximation of the first high- 
frequency component from the second low-frequency and high- 
frequency pair in the second image direction, and by then 
30 reconstructing the digital output video frame from the 
approximations of the first low-frequency component and the 
first high-frequency component in the first image 
direction. 
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31. The system of claim 30, wherein the compressor 
circuitry further comprises: 

main controller circuitry for controlling the 
operation of the decomposing circuitry, so that the low- 
5 frequency component of said first pair may be repetitively 
provided to the decomposing circuitry for further 
decomposing. 

and wherein the decompressor circuitry further 
comprises : 

10 main controller circuitry, for controlling the 

, operation of the reconstructing circuitry so that the 
result of the reconstructing may be repetitively operated 
upon by the reconstructing circuitry, depending upon the 
number of times the decomposing circuitry repetitively 

15 decomposed the digital input video frame. 

32. The system of claim 30, wherein said compressor 
circuitry further comprises: 

lossless compression circuitry, having an input 

20 coupled to said quantization circuitry, for performing 
lossless compression of the first and second low-frequency 
and high-frequency pairs, prior to presenting the 
decomposed frames of digital video image information to the 
communications network; 

25 and wherein said decompressor circuitry further 

comprises: 

lossless decompression circuitry, having an input 
coupled to the communications network for performing 
lossless decompression of the digital video image 
30 information received therefrom. 

33. A system for displaying compressed video image 
data , comprising : 

a memory for storing a plurality of compressed 
35 video frames; 
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decompressor circuitry for reconstructing each of 
said plurality of compressed frames in a first image 
direction using a boundary-spline-wavelet, and for then 
further reconstructing each of said plurality of compressed 
5 frames in a second image direction using a boundary-spline- 
wavelet; and 

a video display , coupled to said decompressor 
circuitry, for displaying each of said plurality of 
reconstructed frames. 

10 

34. The system of claim 33, wherein said memory is 
coupled to a communications network. 

35. The system of claim 33, wherein said decompressor 
15 circuitry comprises: 

a digital matrix processor coupled to said 
memory, for reconstructing each of said plurality of frames 
using precalculated matrices. 

20 36 • The system of claim 33, wherein each of said 

plurality of frames are stored in said memory in a form of 
the type comprising quantized first and second pairs of 
low- frequency and high-frequency components, said first 
pair of low-frequency and high-frequency components 

25 corresponding to the decomposition, in the first direction, 
of a low-frequency decomposition in the second direction of 
a video image frame, and said second pair of low- frequency 
and high-frequency components corresponding to the 
decomposition, in tfhe first direction, of a high-frequency 

30 decomposition in the second direction of the video image 
frame. 

37. The system of claim 36, wherein each of said 
plurality of frames are stored in said memory in a form 
35 including the lossless compression of said quantized first 
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and second pairs of low-frequency and high-frequency 
components ; 

and further comprising: 

lossless decompression circuitry, coupled between 
5 said memory and said decompressor circuitry, for performing 
lossless decompression of each of said plurality of frames 
prior to the reconstruction thereof by said decompressor 
circuitry, 

10 38. The system of claim 36, wherein said first and 

second pairs of low-frequency and high-frequency components 
correspond to the results of repetitively decomposed 
components of a video image; 
and further comprising: 

15 main controller circuitry, for controlling the 

operation of the decompressing circuitry so that the result 
of the reconstructing may be repetitively operated upon by 
the decompressing circuitry according to the number of 
times that the stored frames were repetitively decomposed. 

20 

39. The system of claim 33, wherein each of said 
plurality of frames stored in said memory correspond to a 
plurality of compressed image blocks; 

and wherein said decompressor circuitry reconstructs 

25 each of the compressed image blocks for each of said 
plurality of compressed frames in the first image direction 
using a boundary-spline-wavelet, and for then further 
reconstructing each of the compressed image blocks for said 
plurality of compressed frames in the second image 

30 direction using a boundary-spline-wavelet. 
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